2021-09-16T19:20:02.467 INFO:root:teuthology version: 1.1.0-63133ca9 2021-09-16T19:20:02.468 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi} --owner scheduled_mbenjamin@shatner --name mbenjamin-2021-09-16_15:12:17-rgw-wip-rgwadminops-fsid-distro-basic-smithi --archive /home/teuthworker/archive/mbenjamin-2021-09-16_15:12:17-rgw-wip-rgwadminops-fsid-distro-basic-smithi/6393866 --verbose -- /home/teuthworker/archive/mbenjamin-2021-09-16_15:12:17-rgw-wip-rgwadminops-fsid-distro-basic-smithi/6393866/orig.config.yaml 2021-09-16T19:20:02.507 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-09-16T19:20:02.561 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mbenjamin-2021-09-16_15:12:17-rgw-wip-rgwadminops-fsid-distro-basic-smithi/6393866 branch: wip-rgwadminops-fsid description: rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi} email: null first_in_suite: false job_id: '6393866' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: mbenjamin-2021-09-16_15:12:17-rgw-wip-rgwadminops-fsid-distro-basic-smithi nuke-on-error: true os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-rgwadminops-fsid ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: bc749ef43279601f3e9305d4ad8723d440abaa66 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: bc749ef43279601f3e9305d4ad8723d440abaa66 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 rgw-multisite-tests: args: - tests.py - tests_ps.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-rgw-sts-op-logs-plus-perl sha1: 4e2e64ee99b6cbb34351964aad0f9df4874dea6d owner: scheduled_mbenjamin@shatner priority: 70 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: bc749ef43279601f3e9305d4ad8723d440abaa66 sleep_before_teardown: 0 suite: rgw suite_branch: wip-rgw-sts-op-logs-plus-perl suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_4e2e64ee99b6cbb34351964aad0f9df4874dea6d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 4e2e64ee99b6cbb34351964aad0f9df4874dea6d targets: smithi109.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDit7nKIpkndJnsa95XTFOPPxPMn5UoSxviofbE8fdv6otd42BKcm+tP55ory6N3+MkrJLaUuqFz++RRoIZKIWDcrH84eNqA4+6QouaQAgu+pE/Up1idQaX1XOmi+sPHX4sZTmcUJD1EAgS5otgTbEcMvcjdsA1HCJTSXjjBYyOFbiiw0GTgCn3ur6vmRzf55VaJX4KQYJc5ncAxgHHtVZmtUmKNRAbNqRWU0dALAwPbZPMXDoiL06z2mMoOQiEOPB9HeNfTuiumQhor88FAuB4QwswCJIUn7kWcdd+Q50B8C2xcULqP19KDmOPzZ6VQ+k5D2oMzY3hXf5NCrXZBb4OyOm8kiG/L2T4dva3ZA0TY/Gd7+dTeiob9BbjfRCf/aSc5QiZSSp5hcf2iUdcP1EeaDW6bNvp7A1u2QrP7truKxgtXs1J6w62p4JPwPuDGBWj6iN6wgVSmh7Cl6wnLGkgXJS3BA6DvvIQVZ4Mm/q5sTNPXajC8qAjF7gOaQXhO4s= smithi118.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/mv+u+EjO7vwGqKRrKc1/4hE/xmnvsVd0Dz7wFkd88PDWJbemZ0yiW25IgARYTm1yKTAsbsE6T9o8nL9St/CSjrFYHTLfgL4e1XJ7JeNSOFEGMLye3FRvMZ+fBC3rYHO9i6zAP9PGrBtAxXfnx6TxfrPHuYM/uvHskbIccY0RgCC8ZtBtH5Ank0H+/eRfabQU0KlqamP1bvV7gDoSkhaGM7WJg+q9THZC0j3zvaN+q7fOQnjjvUPyClYttfIx8bYhaG7yGfxxuLzX+8bnQWJE4x5Hz2/aUl+9wgwvuK68Bt8mZebdgTj6gzWDZ9FEn5bfSTwXfP5rQmyqQt95CcEZHdS7KKFTP9tN8dwaJVxZDtbqqh/PsfuOo7L1ZDZQUINeGyV0qozcyHbM2O+GKMN4dGPCj2ByfmhAwn6ktXmQ/HOeNbZ8jTVpodUurhwKCd2OU2+dR1Y/Zu5m94V1g6ona4db93N+Z1l3Ry43ou9WOE7bwdhTGYNzeGznkFdq8lM= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master teuthology_sha1: 63133ca92c299bd23f40f94d370cf5381b0b1313 timestamp: 2021-09-16_15:12:17 tube: smithi user: mbenjamin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.30581 2021-09-16T19:20:02.562 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_4e2e64ee99b6cbb34351964aad0f9df4874dea6d/qa; will attempt to use it 2021-09-16T19:20:02.563 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_4e2e64ee99b6cbb34351964aad0f9df4874dea6d/qa/tasks 2021-09-16T19:20:02.563 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-09-16T19:20:02.565 INFO:teuthology.task.internal:Checking packages... 2021-09-16T19:20:02.585 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash 'bc749ef43279601f3e9305d4ad8723d440abaa66' 2021-09-16T19:20:02.586 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-09-16T19:20:02.586 INFO:teuthology.packaging:ref: None 2021-09-16T19:20:02.587 INFO:teuthology.packaging:tag: None 2021-09-16T19:20:02.587 INFO:teuthology.packaging:branch: wip-rgwadminops-fsid 2021-09-16T19:20:02.587 INFO:teuthology.packaging:sha1: bc749ef43279601f3e9305d4ad8723d440abaa66 2021-09-16T19:20:02.588 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-rgwadminops-fsid 2021-09-16T19:20:02.751 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-7730.gbc749ef4 2021-09-16T19:20:02.751 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-09-16T19:20:02.759 INFO:teuthology.task.internal:no buildpackages task found 2021-09-16T19:20:02.760 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-09-16T19:20:02.800 INFO:teuthology.task.internal:Saving configuration 2021-09-16T19:20:02.815 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-09-16T19:20:02.824 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-16T19:20:02.843 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-09-16 19:14:42.235862', 'description': '/home/teuthworker/archive/mbenjamin-2021-09-16_15:12:17-rgw-wip-rgwadminops-fsid-distro-basic-smithi/6393866', 'locked_by': 'scheduled_mbenjamin@shatner', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBFC7ZT+MYOl+0I8nnr8kb7zBjakof8hRgkxDf+CMT9OUMuJX7QBeFhws5wmX3HSjuSmAntyng512wmSSAro+MI=', 'os_version': '8.4', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi109.front.sepia.ceph.com'} 2021-09-16T19:20:02.866 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-09-16 19:14:42.235885', 'description': '/home/teuthworker/archive/mbenjamin-2021-09-16_15:12:17-rgw-wip-rgwadminops-fsid-distro-basic-smithi/6393866', 'locked_by': 'scheduled_mbenjamin@shatner', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDYnMwMnFTBw+u5M67N28iQriy+5qkyRpR2tiiaNeIwEI9y2U5AfZfUbn4MqiV867iMMsvBNJiwdngGPkSR2rlY=', 'os_version': '8.4', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi118.front.sepia.ceph.com'} 2021-09-16T19:20:02.867 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-09-16T19:20:02.874 INFO:teuthology.task.internal:roles: ubuntu@smithi109.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2021-09-16T19:20:02.875 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2021-09-16T19:20:02.875 INFO:teuthology.run_tasks:Running task console_log... 2021-09-16T19:20:02.993 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa473f4eea0>, signals=[15]) 2021-09-16T19:20:02.994 INFO:teuthology.run_tasks:Running task internal.connect... 2021-09-16T19:20:03.002 INFO:teuthology.task.internal:Opening connections... 2021-09-16T19:20:03.003 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi109.front.sepia.ceph.com 2021-09-16T19:20:03.005 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi109.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-16T19:20:03.074 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2021-09-16T19:20:03.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-16T19:20:03.146 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-09-16T19:20:03.154 DEBUG:teuthology.orchestra.run.smithi109:> uname -m 2021-09-16T19:20:03.170 INFO:teuthology.orchestra.run.smithi109.stdout:x86_64 2021-09-16T19:20:03.171 DEBUG:teuthology.orchestra.run.smithi109:> cat /etc/os-release 2021-09-16T19:20:03.225 INFO:teuthology.orchestra.run.smithi109.stdout:NAME="Red Hat Enterprise Linux" 2021-09-16T19:20:03.225 INFO:teuthology.orchestra.run.smithi109.stdout:VERSION="8.4 (Ootpa)" 2021-09-16T19:20:03.226 INFO:teuthology.orchestra.run.smithi109.stdout:ID="rhel" 2021-09-16T19:20:03.226 INFO:teuthology.orchestra.run.smithi109.stdout:ID_LIKE="fedora" 2021-09-16T19:20:03.226 INFO:teuthology.orchestra.run.smithi109.stdout:VERSION_ID="8.4" 2021-09-16T19:20:03.227 INFO:teuthology.orchestra.run.smithi109.stdout:PLATFORM_ID="platform:el8" 2021-09-16T19:20:03.227 INFO:teuthology.orchestra.run.smithi109.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2021-09-16T19:20:03.227 INFO:teuthology.orchestra.run.smithi109.stdout:ANSI_COLOR="0;31" 2021-09-16T19:20:03.228 INFO:teuthology.orchestra.run.smithi109.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2021-09-16T19:20:03.228 INFO:teuthology.orchestra.run.smithi109.stdout:HOME_URL="https://www.redhat.com/" 2021-09-16T19:20:03.228 INFO:teuthology.orchestra.run.smithi109.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2021-09-16T19:20:03.228 INFO:teuthology.orchestra.run.smithi109.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-09-16T19:20:03.228 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:20:03.229 INFO:teuthology.orchestra.run.smithi109.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-09-16T19:20:03.229 INFO:teuthology.orchestra.run.smithi109.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2021-09-16T19:20:03.229 INFO:teuthology.orchestra.run.smithi109.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-09-16T19:20:03.229 INFO:teuthology.orchestra.run.smithi109.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2021-09-16T19:20:03.231 INFO:teuthology.lock.ops:Updating smithi109.front.sepia.ceph.com on lock server 2021-09-16T19:20:03.248 DEBUG:teuthology.orchestra.run.smithi118:> uname -m 2021-09-16T19:20:03.264 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2021-09-16T19:20:03.264 DEBUG:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2021-09-16T19:20:03.318 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="Red Hat Enterprise Linux" 2021-09-16T19:20:03.319 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="8.4 (Ootpa)" 2021-09-16T19:20:03.319 INFO:teuthology.orchestra.run.smithi118.stdout:ID="rhel" 2021-09-16T19:20:03.319 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="fedora" 2021-09-16T19:20:03.319 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="8.4" 2021-09-16T19:20:03.319 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el8" 2021-09-16T19:20:03.320 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2021-09-16T19:20:03.320 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31" 2021-09-16T19:20:03.320 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2021-09-16T19:20:03.320 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://www.redhat.com/" 2021-09-16T19:20:03.321 INFO:teuthology.orchestra.run.smithi118.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2021-09-16T19:20:03.321 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-09-16T19:20:03.321 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:20:03.321 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-09-16T19:20:03.321 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2021-09-16T19:20:03.322 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-09-16T19:20:03.322 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2021-09-16T19:20:03.322 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2021-09-16T19:20:03.337 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-09-16T19:20:03.347 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-09-16T19:20:03.367 INFO:teuthology.task.internal:Checking for old test directory... 2021-09-16T19:20:03.367 DEBUG:teuthology.orchestra.run.smithi109:> test '!' -e /home/ubuntu/cephtest 2021-09-16T19:20:03.370 DEBUG:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2021-09-16T19:20:03.384 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-09-16T19:20:03.393 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-09-16T19:20:03.393 DEBUG:teuthology.orchestra.run.smithi109:> test -z $(ls -A /var/lib/ceph) 2021-09-16T19:20:03.425 DEBUG:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2021-09-16T19:20:03.449 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-09-16T19:20:03.490 INFO:teuthology.run_tasks:Running task kernel... 2021-09-16T19:20:03.509 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2021-09-16T19:20:03.509 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-09-16T19:20:03.510 DEBUG:teuthology.orchestra.run.smithi109:> uname -r 2021-09-16T19:20:03.525 INFO:teuthology.orchestra.run.smithi109.stdout:4.18.0-305.el8.x86_64 2021-09-16T19:20:03.526 INFO:teuthology.task.kernel:Running kernel on smithi109: 4.18.0-305.el8.x86_64 2021-09-16T19:20:03.526 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum install -y kernel 2021-09-16T19:20:04.911 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:20:04.911 INFO:teuthology.orchestra.run.smithi109.stdout:Unable to read consumer identity 2021-09-16T19:20:05.037 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:20:05.037 INFO:teuthology.orchestra.run.smithi109.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-16T19:20:05.037 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:20:07.763 INFO:teuthology.orchestra.run.smithi109.stdout:Extra Packages for Enterprise Linux 6.6 MB/s | 10 MB 00:01 2021-09-16T19:20:09.797 INFO:teuthology.orchestra.run.smithi109.stdout:lab-extras 4.0 MB/s | 24 kB 00:00 2021-09-16T19:20:10.392 INFO:teuthology.orchestra.run.smithi109.stdout:Package kernel-4.18.0-305.el8.x86_64 is already installed. 2021-09-16T19:20:10.502 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:20:10.504 INFO:teuthology.orchestra.run.smithi109.stdout:Nothing to do. 2021-09-16T19:20:10.504 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:20:10.653 DEBUG:teuthology.orchestra.run.smithi109:> echo no | sudo yum reinstall kernel || true 2021-09-16T19:20:10.970 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:20:10.970 INFO:teuthology.orchestra.run.smithi109.stdout:Unable to read consumer identity 2021-09-16T19:20:11.008 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:20:11.009 INFO:teuthology.orchestra.run.smithi109.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-16T19:20:11.009 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:20:11.188 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:02 ago on Thu 16 Sep 2021 07:20:09 PM UTC. 2021-09-16T19:20:11.305 INFO:teuthology.orchestra.run.smithi109.stdout:Installed package kernel-4.18.0-305.el8.x86_64 (from anaconda) not available. 2021-09-16T19:20:11.382 INFO:teuthology.orchestra.run.smithi109.stderr:Error: No packages marked for reinstall. 2021-09-16T19:20:11.426 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum reinstall -y kernel || true 2021-09-16T19:20:11.738 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:20:11.738 INFO:teuthology.orchestra.run.smithi109.stdout:Unable to read consumer identity 2021-09-16T19:20:11.787 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:20:11.787 INFO:teuthology.orchestra.run.smithi109.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-16T19:20:11.788 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:20:11.960 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:02 ago on Thu 16 Sep 2021 07:20:09 PM UTC. 2021-09-16T19:20:12.083 INFO:teuthology.orchestra.run.smithi109.stdout:Installed package kernel-4.18.0-305.el8.x86_64 (from anaconda) not available. 2021-09-16T19:20:12.148 INFO:teuthology.orchestra.run.smithi109.stderr:Error: No packages marked for reinstall. 2021-09-16T19:20:12.190 DEBUG:teuthology.orchestra.run.smithi109:> rpm -q kernel --last | head -n 1 2021-09-16T19:20:12.258 INFO:teuthology.orchestra.run.smithi109.stdout:kernel-4.18.0-305.el8.x86_64 Thu 20 May 2021 08:07:00 PM UTC 2021-09-16T19:20:12.259 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.el8.x86_64 2021-09-16T19:20:12.260 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-09-16T19:20:12.260 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2021-09-16T19:20:12.277 INFO:teuthology.orchestra.run.smithi118.stdout:4.18.0-305.el8.x86_64 2021-09-16T19:20:12.278 INFO:teuthology.task.kernel:Running kernel on smithi118: 4.18.0-305.el8.x86_64 2021-09-16T19:20:12.278 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum install -y kernel 2021-09-16T19:20:13.678 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:20:13.679 INFO:teuthology.orchestra.run.smithi118.stdout:Unable to read consumer identity 2021-09-16T19:20:13.806 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:20:13.806 INFO:teuthology.orchestra.run.smithi118.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-16T19:20:13.807 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:20:16.155 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 8.5 MB/s | 10 MB 00:01 2021-09-16T19:20:18.194 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 1.2 MB/s | 24 kB 00:00 2021-09-16T19:20:18.740 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-4.18.0-305.el8.x86_64 is already installed. 2021-09-16T19:20:18.853 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:20:18.853 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2021-09-16T19:20:18.854 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:20:18.999 DEBUG:teuthology.orchestra.run.smithi118:> echo no | sudo yum reinstall kernel || true 2021-09-16T19:20:19.313 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:20:19.313 INFO:teuthology.orchestra.run.smithi118.stdout:Unable to read consumer identity 2021-09-16T19:20:19.356 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:20:19.356 INFO:teuthology.orchestra.run.smithi118.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-16T19:20:19.356 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:20:19.531 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:01 ago on Thu 16 Sep 2021 07:20:18 PM UTC. 2021-09-16T19:20:19.680 INFO:teuthology.orchestra.run.smithi118.stdout:Installed package kernel-4.18.0-305.el8.x86_64 (from anaconda) not available. 2021-09-16T19:20:19.725 INFO:teuthology.orchestra.run.smithi118.stderr:Error: No packages marked for reinstall. 2021-09-16T19:20:19.768 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum reinstall -y kernel || true 2021-09-16T19:20:20.080 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:20:20.081 INFO:teuthology.orchestra.run.smithi118.stdout:Unable to read consumer identity 2021-09-16T19:20:20.119 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:20:20.119 INFO:teuthology.orchestra.run.smithi118.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-16T19:20:20.119 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:20:20.291 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:02 ago on Thu 16 Sep 2021 07:20:18 PM UTC. 2021-09-16T19:20:20.438 INFO:teuthology.orchestra.run.smithi118.stdout:Installed package kernel-4.18.0-305.el8.x86_64 (from anaconda) not available. 2021-09-16T19:20:20.473 INFO:teuthology.orchestra.run.smithi118.stderr:Error: No packages marked for reinstall. 2021-09-16T19:20:20.515 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel --last | head -n 1 2021-09-16T19:20:20.579 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-4.18.0-305.el8.x86_64 Thu 20 May 2021 08:07:00 PM UTC 2021-09-16T19:20:20.581 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.el8.x86_64 2021-09-16T19:20:20.581 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-09-16T19:20:20.581 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2021-09-16T19:20:20.582 DEBUG:teuthology.orchestra.run.smithi109:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-09-16T19:20:20.634 INFO:teuthology.orchestra.run.smithi109.stdout:ttyS1 2021-09-16T19:20:20.647 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2021-09-16T19:20:20.648 DEBUG:teuthology.orchestra.run.smithi118:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-09-16T19:20:20.725 INFO:teuthology.orchestra.run.smithi118.stdout:ttyS1 2021-09-16T19:20:20.737 INFO:teuthology.run_tasks:Running task internal.base... 2021-09-16T19:20:20.758 INFO:teuthology.task.internal:Creating test directory... 2021-09-16T19:20:20.758 DEBUG:teuthology.orchestra.run.smithi109:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-09-16T19:20:20.762 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-09-16T19:20:20.778 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-09-16T19:20:20.801 INFO:teuthology.run_tasks:Running task internal.archive... 2021-09-16T19:20:20.825 INFO:teuthology.task.internal:Creating archive directory... 2021-09-16T19:20:20.826 DEBUG:teuthology.orchestra.run.smithi109:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-09-16T19:20:20.829 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-09-16T19:20:20.875 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-09-16T19:20:20.887 INFO:teuthology.task.internal:Enabling coredump saving... 2021-09-16T19:20:20.888 DEBUG:teuthology.orchestra.run.smithi109:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-09-16T19:20:20.909 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-09-16T19:20:20.944 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-16T19:20:20.945 INFO:teuthology.orchestra.run.smithi109.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-16T19:20:20.954 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-16T19:20:20.957 INFO:teuthology.orchestra.run.smithi109.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-16T19:20:20.960 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-09-16T19:20:20.984 INFO:teuthology.task.internal:Configuring sudo... 2021-09-16T19:20:20.985 DEBUG:teuthology.orchestra.run.smithi109:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-09-16T19:20:21.008 DEBUG:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-09-16T19:20:21.032 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-09-16T19:20:21.045 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-09-16T19:20:21.046 DEBUG:teuthology.orchestra.run.smithi109:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-09-16T19:20:21.072 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-09-16T19:20:21.086 DEBUG:teuthology.orchestra.run.smithi109:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-16T19:20:21.161 DEBUG:teuthology.orchestra.run.smithi109:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-16T19:20:21.222 DEBUG:teuthology.orchestra.run.smithi109:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-16T19:20:21.298 DEBUG:teuthology.orchestra.run.smithi109:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-16T19:20:21.350 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:20:21.350 DEBUG:teuthology.orchestra.run.smithi109:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-09-16T19:20:21.413 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-16T19:20:21.447 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-16T19:20:21.508 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-16T19:20:21.583 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-16T19:20:21.628 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:20:21.629 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-09-16T19:20:21.692 DEBUG:teuthology.orchestra.run.smithi109:> sudo service rsyslog restart 2021-09-16T19:20:21.695 DEBUG:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2021-09-16T19:20:21.751 INFO:teuthology.orchestra.run.smithi109.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-09-16T19:20:21.777 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-09-16T19:20:22.072 INFO:teuthology.run_tasks:Running task internal.timer... 2021-09-16T19:20:22.084 INFO:teuthology.task.internal:Starting timer... 2021-09-16T19:20:22.084 INFO:teuthology.run_tasks:Running task pcp... 2021-09-16T19:20:22.136 INFO:teuthology.run_tasks:Running task selinux... 2021-09-16T19:20:22.160 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-09-16T19:20:22.161 DEBUG:teuthology.orchestra.run.smithi109:> sudo service auditd rotate 2021-09-16T19:20:22.257 INFO:teuthology.orchestra.run.smithi109.stdout:Rotating logs: [ OK ] 2021-09-16T19:20:22.259 DEBUG:teuthology.orchestra.run.smithi118:> sudo service auditd rotate 2021-09-16T19:20:22.331 INFO:teuthology.orchestra.run.smithi118.stdout:Rotating logs: [ OK ] 2021-09-16T19:20:22.332 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-09-16T19:20:22.333 DEBUG:teuthology.orchestra.run.smithi109:> /usr/sbin/getenforce 2021-09-16T19:20:22.352 INFO:teuthology.orchestra.run.smithi109.stdout:Permissive 2021-09-16T19:20:22.353 DEBUG:teuthology.orchestra.run.smithi118:> /usr/sbin/getenforce 2021-09-16T19:20:22.380 INFO:teuthology.orchestra.run.smithi118.stdout:Permissive 2021-09-16T19:20:22.380 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi109.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi118.front.sepia.ceph.com': 'permissive'} 2021-09-16T19:20:22.381 DEBUG:teuthology.orchestra.run.smithi109:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-09-16T19:20:22.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-16T19:20:22.419 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-09-16T19:20:22.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-16T19:20:22.444 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-09-16T19:20:22.445 DEBUG:teuthology.orchestra.run.smithi109:> sudo /usr/sbin/setenforce permissive 2021-09-16T19:20:22.485 DEBUG:teuthology.orchestra.run.smithi118:> sudo /usr/sbin/setenforce permissive 2021-09-16T19:20:22.516 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-09-16T19:20:22.528 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-09-16T19:20:22.528 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-09-16T19:20:22.546 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-09-16T19:20:22.548 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi109.front.sepia.ceph.com,smithi118.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-09-16T19:20:22.915 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-09-16T19:20:23.715 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-09-16T19:20:26.598 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-09-16T19:20:26.623 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:26 +0000 (0:00:00.040) 0:00:00.040 **** 2021-09-16T19:20:26.636 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:26 +0000 (0:00:00.013) 0:00:00.054 **** 2021-09-16T19:20:28.072 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi109.front.sepia.ceph.com] 2021-09-16T19:20:28.119 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:28 +0000 (0:00:01.483) 0:00:01.537 **** 2021-09-16T19:20:28.212 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] 2021-09-16T19:20:28.264 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:28 +0000 (0:00:00.145) 0:00:01.682 **** 2021-09-16T19:20:28.386 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-09-16T19:20:28.388 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-16T19:20:28.411 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:28 +0000 (0:00:00.147) 0:00:01.829 **** 2021-09-16T19:20:28.450 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-16T19:20:28.472 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:28 +0000 (0:00:00.060) 0:00:01.890 **** 2021-09-16T19:20:28.779 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-09-16T19:20:28.781 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-09-16T19:20:28.802 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:28 +0000 (0:00:00.330) 0:00:02.220 **** 2021-09-16T19:20:28.809 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-09-16T19:20:28.832 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:28 +0000 (0:00:00.030) 0:00:02.250 **** 2021-09-16T19:20:28.861 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-09-16T19:20:28.863 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:20:28.885 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:28 +0000 (0:00:00.052) 0:00:02.303 **** 2021-09-16T19:20:28.903 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:20:28.926 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:28 +0000 (0:00:00.040) 0:00:02.343 **** 2021-09-16T19:20:29.141 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi118.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-16T19:20:29.143 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-16T19:20:29.163 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:29 +0000 (0:00:00.237) 0:00:02.581 **** 2021-09-16T19:20:29.181 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:29 +0000 (0:00:00.018) 0:00:02.599 **** 2021-09-16T19:20:30.036 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-09-16T19:20:30.038 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1631820029.2233694-24117-98165322567916/source", "state": "file", "uid": 0} changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1631820029.2060692-24115-261872065632407/source", "state": "file", "uid": 0} 2021-09-16T19:20:30.058 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:30 +0000 (0:00:00.877) 0:00:03.476 **** 2021-09-16T19:20:30.079 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:30 +0000 (0:00:00.020) 0:00:03.497 **** 2021-09-16T19:20:30.463 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-09-16T19:20:30.465 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-09-16T19:20:30.467 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-09-16T19:20:30.489 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-09-16T19:20:30.491 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-09-16T19:20:30.564 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-09-16T19:20:30.565 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-09-16T19:20:30.578 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-09-16T19:20:30.579 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:30 +0000 (0:00:00.498) 0:00:03.996 **** 2021-09-16T19:20:31.110 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] 2021-09-16T19:20:31.128 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] 2021-09-16T19:20:31.145 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** Thursday 16 September 2021 19:20:31 +0000 (0:00:00.566) 0:00:04.563 **** 2021-09-16T19:20:31.171 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-09-16T19:20:31.182 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-09-16T19:20:31.187 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-09-16T19:20:31.341 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:31 +0000 (0:00:00.196) 0:00:04.759 **** 2021-09-16T19:20:31.361 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:31 +0000 (0:00:00.019) 0:00:04.779 **** 2021-09-16T19:20:31.978 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi109.front.sepia.ceph.com] 2021-09-16T19:20:32.008 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] 2021-09-16T19:20:32.037 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Thursday 16 September 2021 19:20:32 +0000 (0:00:00.675) 0:00:05.455 **** 2021-09-16T19:20:32.081 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:32 +0000 (0:00:00.044) 0:00:05.499 **** 2021-09-16T19:20:32.709 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-09-16T19:20:32.711 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.184649", "end": "2021-09-16 19:20:32.681553", "rc": 0, "start": "2021-09-16 19:20:32.496904", "stderr": "", "stderr_lines": [], "stdout": "39f530053dd0894c844c464d403644cc64956570", "stdout_lines": ["39f530053dd0894c844c464d403644cc64956570"]} 2021-09-16T19:20:32.732 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:32 +0000 (0:00:00.649) 0:00:06.149 **** 2021-09-16T19:20:32.738 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.193521", "end": "2021-09-16 19:20:32.692738", "rc": 0, "start": "2021-09-16 19:20:32.499217", "stderr": "", "stderr_lines": [], "stdout": "39f530053dd0894c844c464d403644cc64956570", "stdout_lines": ["39f530053dd0894c844c464d403644cc64956570"]} 2021-09-16T19:20:32.762 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:32 +0000 (0:00:00.031) 0:00:06.180 **** 2021-09-16T19:20:32.936 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-09-16T19:20:32.938 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.011013", "end": "2021-09-16 19:20:32.913314", "failed_when_result": false, "rc": 0, "start": "2021-09-16 19:20:32.902301", "stderr": "", "stderr_lines": [], "stdout": "b2a91f5f527dcc382f8ed49697328a42af627ee8", "stdout_lines": ["b2a91f5f527dcc382f8ed49697328a42af627ee8"]} 2021-09-16T19:20:32.959 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:32 +0000 (0:00:00.196) 0:00:06.376 **** 2021-09-16T19:20:32.967 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.010070", "end": "2021-09-16 19:20:32.942017", "failed_when_result": false, "rc": 0, "start": "2021-09-16 19:20:32.931947", "stderr": "", "stderr_lines": [], "stdout": "b2a91f5f527dcc382f8ed49697328a42af627ee8", "stdout_lines": ["b2a91f5f527dcc382f8ed49697328a42af627ee8"]} 2021-09-16T19:20:32.991 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:32 +0000 (0:00:00.032) 0:00:06.409 **** 2021-09-16T19:20:33.028 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-09-16T19:20:33.030 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:20:33.051 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:33 +0000 (0:00:00.060) 0:00:06.469 **** 2021-09-16T19:20:33.078 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:20:33.101 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:33 +0000 (0:00:00.049) 0:00:06.519 **** 2021-09-16T19:20:33.122 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-09-16T19:20:33.124 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:20:33.145 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:33 +0000 (0:00:00.043) 0:00:06.563 **** 2021-09-16T19:20:33.182 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:20:33.204 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:33 +0000 (0:00:00.059) 0:00:06.622 **** 2021-09-16T19:20:33.222 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:20:33.245 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:33 +0000 (0:00:00.040) 0:00:06.663 **** 2021-09-16T19:20:33.272 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:20:33.296 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:33 +0000 (0:00:00.050) 0:00:06.714 **** 2021-09-16T19:20:33.314 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-09-16T19:20:33.316 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:20:33.337 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:33 +0000 (0:00:00.041) 0:00:06.755 **** 2021-09-16T19:20:33.365 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:20:33.389 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:33 +0000 (0:00:00.051) 0:00:06.807 **** 2021-09-16T19:20:33.407 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-09-16T19:20:33.409 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:20:33.431 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:33 +0000 (0:00:00.041) 0:00:06.849 **** 2021-09-16T19:20:33.459 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:20:33.483 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:33 +0000 (0:00:00.052) 0:00:06.901 **** 2021-09-16T19:20:33.669 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-09-16T19:20:33.670 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-16T19:20:33.694 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:33 +0000 (0:00:00.210) 0:00:07.112 **** 2021-09-16T19:20:33.724 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-16T19:20:33.747 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:20:33 +0000 (0:00:00.052) 0:00:07.165 **** 2021-09-16T19:20:33.975 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-09-16T19:20:33.977 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-09-16T19:20:34.019 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-09-16T19:20:34.176 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-09-16T19:20:34.219 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-09-16T19:20:34.365 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-09-16T19:20:34.419 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-09-16T19:20:34.566 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-09-16T19:20:34.609 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-09-16T19:20:34.756 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-09-16T19:20:34.810 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-09-16T19:20:34.957 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-09-16T19:20:34.999 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-09-16T19:20:35.146 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-09-16T19:20:35.200 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-09-16T19:20:35.336 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-09-16T19:20:35.390 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-09-16T19:20:35.537 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-09-16T19:20:35.580 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-09-16T19:20:35.727 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-09-16T19:20:35.781 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-09-16T19:20:35.927 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-09-16T19:20:35.980 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-09-16T19:20:36.116 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-09-16T19:20:36.180 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-09-16T19:20:36.317 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-09-16T19:20:36.374 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-09-16T19:20:36.512 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-09-16T19:20:36.565 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-09-16T19:20:36.701 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-09-16T19:20:36.764 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-09-16T19:20:36.901 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-09-16T19:20:36.954 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-09-16T19:20:37.091 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-09-16T19:20:37.157 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-09-16T19:20:37.282 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-09-16T19:20:37.346 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-09-16T19:20:37.482 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-09-16T19:20:37.547 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-09-16T19:20:37.676 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-09-16T19:20:37.741 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-09-16T19:20:37.870 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-09-16T19:20:37.934 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-09-16T19:20:38.070 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-09-16T19:20:38.124 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-09-16T19:20:38.261 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-09-16T19:20:38.315 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-09-16T19:20:38.461 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-09-16T19:20:38.514 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-09-16T19:20:38.650 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-09-16T19:20:38.713 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-09-16T19:20:38.860 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-09-16T19:20:38.904 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-09-16T19:20:39.040 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-09-16T19:20:39.098 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-09-16T19:20:39.240 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-09-16T19:20:39.284 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-09-16T19:20:39.430 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-09-16T19:20:39.494 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-09-16T19:20:39.621 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-09-16T19:20:39.674 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-09-16T19:20:39.824 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-09-16T19:20:39.867 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-09-16T19:20:40.015 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-09-16T19:20:40.068 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-09-16T19:20:40.216 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-09-16T19:20:40.270 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-09-16T19:20:40.408 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-09-16T19:20:40.462 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-09-16T19:20:40.609 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-16T19:20:40.663 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-16T19:20:40.800 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-09-16T19:20:40.853 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-09-16T19:20:41.005 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-09-16T19:20:41.056 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-09-16T19:20:41.206 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-09-16T19:20:41.248 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-09-16T19:20:41.392 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-09-16T19:20:41.456 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-09-16T19:20:41.587 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-09-16T19:20:41.651 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-09-16T19:20:41.791 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-09-16T19:20:41.847 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-09-16T19:20:41.975 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-09-16T19:20:42.049 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-09-16T19:20:42.179 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-16T19:20:42.243 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-16T19:20:42.374 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-09-16T19:20:42.441 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-09-16T19:20:42.566 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-09-16T19:20:42.642 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-09-16T19:20:42.768 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-09-16T19:20:42.843 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-09-16T19:20:42.960 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-09-16T19:20:43.045 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-09-16T19:20:43.150 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-09-16T19:20:43.229 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-09-16T19:20:43.344 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-09-16T19:20:43.428 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-09-16T19:20:43.555 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-09-16T19:20:43.629 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-09-16T19:20:43.748 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-09-16T19:20:43.836 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-09-16T19:20:43.952 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-09-16T19:20:44.025 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-09-16T19:20:44.145 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-09-16T19:20:44.219 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-09-16T19:20:44.336 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-09-16T19:20:44.422 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-09-16T19:20:44.529 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-09-16T19:20:44.614 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-09-16T19:20:44.733 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-09-16T19:20:44.817 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-09-16T19:20:44.923 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-09-16T19:20:45.011 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-09-16T19:20:45.118 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-09-16T19:20:45.213 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-09-16T19:20:45.318 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-09-16T19:20:45.415 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-09-16T19:20:45.511 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-16T19:20:45.608 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-16T19:20:45.717 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-09-16T19:20:45.802 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-09-16T19:20:45.907 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-09-16T19:20:46.002 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-09-16T19:20:46.108 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-09-16T19:20:46.193 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-09-16T19:20:46.330 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-09-16T19:20:46.439 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-09-16T19:20:46.520 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-09-16T19:20:46.637 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-09-16T19:20:46.724 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-09-16T19:20:46.840 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-09-16T19:20:46.914 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-09-16T19:20:47.031 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-09-16T19:20:47.106 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-09-16T19:20:47.230 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-09-16T19:20:47.316 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-09-16T19:20:47.436 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-09-16T19:20:47.506 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-09-16T19:20:47.633 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-09-16T19:20:47.699 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-09-16T19:20:47.829 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-09-16T19:20:47.894 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-09-16T19:20:48.031 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-09-16T19:20:48.107 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-09-16T19:20:48.223 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-09-16T19:20:48.297 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-09-16T19:20:48.423 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-09-16T19:20:48.499 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-09-16T19:20:48.622 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-09-16T19:20:48.697 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-09-16T19:20:48.811 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-09-16T19:20:48.895 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-09-16T19:20:49.012 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-09-16T19:20:49.085 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-09-16T19:20:49.201 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-09-16T19:20:49.289 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-09-16T19:20:49.393 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-09-16T19:20:49.477 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-09-16T19:20:49.582 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-09-16T19:20:49.677 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-09-16T19:20:49.782 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-09-16T19:20:49.880 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-09-16T19:20:49.976 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-09-16T19:20:50.070 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-09-16T19:20:50.165 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-09-16T19:20:50.260 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-09-16T19:20:50.365 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-09-16T19:20:50.460 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-09-16T19:20:50.555 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-09-16T19:20:50.661 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-09-16T19:20:50.746 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-09-16T19:20:50.852 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-09-16T19:20:50.946 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-09-16T19:20:51.041 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-09-16T19:20:51.146 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-09-16T19:20:51.240 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-09-16T19:20:51.337 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-09-16T19:20:51.432 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-09-16T19:20:51.528 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-09-16T19:20:51.633 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-09-16T19:20:51.729 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-09-16T19:20:51.824 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-09-16T19:20:51.919 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-09-16T19:20:52.024 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-09-16T19:20:52.119 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-09-16T19:20:52.213 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-09-16T19:20:52.308 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-09-16T19:20:52.413 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-09-16T19:20:52.507 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-09-16T19:20:52.602 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-16T19:20:52.696 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-16T19:20:52.792 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-09-16T19:20:52.898 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-09-16T19:20:52.993 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-09-16T19:20:53.088 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-09-16T19:20:53.194 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-09-16T19:20:53.294 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-09-16T19:20:53.388 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-09-16T19:20:53.482 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-09-16T19:20:53.577 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-09-16T19:20:53.682 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-09-16T19:20:53.777 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-09-16T19:20:53.874 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-09-16T19:20:53.970 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-09-16T19:20:54.076 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-09-16T19:20:54.161 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-09-16T19:20:54.275 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-09-16T19:20:54.360 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-09-16T19:20:54.475 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-09-16T19:20:54.550 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-09-16T19:20:54.665 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-09-16T19:20:54.751 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-09-16T19:20:54.868 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-09-16T19:20:54.941 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-09-16T19:20:55.058 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-09-16T19:20:55.133 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-09-16T19:20:55.249 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-09-16T19:20:55.333 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-09-16T19:20:55.448 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-09-16T19:20:55.532 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-09-16T19:20:55.640 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-09-16T19:20:55.724 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-09-16T19:20:55.840 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-09-16T19:20:55.924 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-09-16T19:20:56.039 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-09-16T19:20:56.113 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-09-16T19:20:56.229 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-09-16T19:20:56.315 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-09-16T19:20:56.430 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-09-16T19:20:56.504 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-09-16T19:20:56.620 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-09-16T19:20:56.707 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-09-16T19:20:56.813 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-09-16T19:20:56.896 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-09-16T19:20:57.015 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-09-16T19:20:57.090 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-09-16T19:20:57.205 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-09-16T19:20:57.297 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-09-16T19:20:57.403 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-09-16T19:20:57.499 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-09-16T19:20:57.605 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-09-16T19:20:57.701 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-09-16T19:20:57.798 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-09-16T19:20:57.893 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-09-16T19:20:58.000 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-09-16T19:20:58.095 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-09-16T19:20:58.194 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-09-16T19:20:58.288 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-09-16T19:20:58.393 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-09-16T19:20:58.478 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-09-16T19:20:58.594 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-09-16T19:20:58.679 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-09-16T19:20:58.784 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-09-16T19:20:58.879 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-09-16T19:20:58.985 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-09-16T19:20:59.070 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-09-16T19:20:59.175 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-09-16T19:20:59.259 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-09-16T19:20:59.365 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-09-16T19:20:59.460 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-09-16T19:20:59.566 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-09-16T19:20:59.661 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-09-16T19:20:59.766 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-09-16T19:20:59.862 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-09-16T19:20:59.947 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-09-16T19:21:01.417 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-09-16T19:21:01.574 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-09-16T19:21:03.281 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-09-16T19:21:03.304 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-09-16T19:21:04.768 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-09-16T19:21:04.887 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-09-16T19:21:06.415 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-09-16T19:21:06.447 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-09-16T19:21:08.009 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-09-16T19:21:08.041 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-09-16T19:21:09.511 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-09-16T19:21:09.607 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-09-16T19:21:11.064 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-09-16T19:21:11.210 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-09-16T19:21:12.625 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-09-16T19:21:12.782 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-09-16T19:21:14.133 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-09-16T19:21:14.363 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-09-16T19:21:15.708 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-09-16T19:21:15.926 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-09-16T19:21:17.241 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-09-16T19:21:17.470 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-09-16T19:21:18.856 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-09-16T19:21:19.043 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-09-16T19:21:20.427 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-09-16T19:21:20.614 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-09-16T19:21:22.035 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-09-16T19:21:22.150 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-09-16T19:21:23.580 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-09-16T19:21:23.696 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-09-16T19:21:25.186 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "name": "melissali", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-09-16T19:21:25.332 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "name": "melissali", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-09-16T19:21:26.785 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-09-16T19:21:26.911 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-09-16T19:21:28.418 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-09-16T19:21:28.532 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-09-16T19:21:30.110 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-09-16T19:21:30.173 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-09-16T19:21:31.742 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10121, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10121} 2021-09-16T19:21:31.873 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:21:31 +0000 (0:00:58.126) 0:01:05.291 **** 2021-09-16T19:21:31.933 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10121, "groups": "sudo", "home": "/home/blaine", "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "name": "blaine", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10121} 2021-09-16T19:21:31.944 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-09-16T19:21:31.969 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:21:31 +0000 (0:00:00.096) 0:01:05.387 **** 2021-09-16T19:21:32.005 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-09-16T19:21:32.078 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:21:32 +0000 (0:00:00.108) 0:01:05.496 **** 2021-09-16T19:21:32.252 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-09-16T19:21:32.254 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-09-16T19:21:32.256 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-09-16T19:21:32.257 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed208 2021-09-16T19:21:32.259 INFO:teuthology.task.ansible.out:2caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": 2021-09-16T19:21:32.261 INFO:teuthology.task.ansible.out: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", 2021-09-16T19:21:32.261 INFO:teuthology.task.ansible.out: "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d 2021-09-16T19:21:32.263 INFO:teuthology.task.ansible.out:3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDe 2021-09-16T19:21:32.265 INFO:teuthology.task.ansible.out:vAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBM 2021-09-16T19:21:32.265 INFO:teuthology.task.ansible.out:gw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmeri 2021-09-16T19:21:32.267 INFO:teuthology.task.ansible.out:cp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361 2021-09-16T19:21:32.269 INFO:teuthology.task.ansible.out:f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9 2021-09-16T19:21:32.270 INFO:teuthology.task.ansible.out:JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": " 2021-09-16T19:21:32.272 INFO:teuthology.task.ansible.out:shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f7 2021-09-16T19:21:32.272 INFO:teuthology.task.ansible.out:0cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6 2021-09-16T19:21:32.274 INFO:teuthology.task.ansible.out:878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovp 2021-09-16T19:21:32.277 INFO:teuthology.task.ansible.out:n": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}]}, "changed": false} 2021-09-16T19:21:32.299 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:21:32 +0000 (0:00:00.221) 0:01:05.717 **** 2021-09-16T19:21:32.307 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-09-16T19:21:32.330 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:21:32 +0000 (0:00:00.031) 0:01:05.748 **** 2021-09-16T19:21:32.547 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-09-16T19:21:32.549 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-09-16T19:21:32.550 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-09-16T19:21:32.552 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed208 2021-09-16T19:21:32.554 INFO:teuthology.task.ansible.out:2caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": 2021-09-16T19:21:32.555 INFO:teuthology.task.ansible.out: "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebb 2021-09-16T19:21:32.556 INFO:teuthology.task.ansible.out:a6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d 2021-09-16T19:21:32.558 INFO:teuthology.task.ansible.out:3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDe 2021-09-16T19:21:32.559 INFO:teuthology.task.ansible.out:vAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBM 2021-09-16T19:21:32.561 INFO:teuthology.task.ansible.out:gw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmeri 2021-09-16T19:21:32.562 INFO:teuthology.task.ansible.out:cp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361 2021-09-16T19:21:32.564 INFO:teuthology.task.ansible.out:f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9 2021-09-16T19:21:32.566 INFO:teuthology.task.ansible.out:JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa15674 2021-09-16T19:21:32.566 INFO:teuthology.task.ansible.out:32aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": " 2021-09-16T19:21:32.568 INFO:teuthology.task.ansible.out:shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6 2021-09-16T19:21:32.570 INFO:teuthology.task.ansible.out:878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovp 2021-09-16T19:21:32.571 INFO:teuthology.task.ansible.out:n": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}]}, "changed": false} 2021-09-16T19:21:32.592 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:21:32 +0000 (0:00:00.261) 0:01:06.010 **** 2021-09-16T19:21:33.450 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* ok: [smithi118.front.sepia.ceph.com] => {"after": "39f530053dd0894c844c464d403644cc64956570", "attempts": 1, "before": "39f530053dd0894c844c464d403644cc64956570", "changed": false, "remote_url_changed": false} 2021-09-16T19:21:33.473 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:21:33 +0000 (0:00:00.880) 0:01:06.890 **** 2021-09-16T19:21:33.501 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"after": "39f530053dd0894c844c464d403644cc64956570", "attempts": 1, "before": "39f530053dd0894c844c464d403644cc64956570", "changed": false, "remote_url_changed": false} 2021-09-16T19:21:33.525 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:21:33 +0000 (0:00:00.052) 0:01:06.943 **** 2021-09-16T19:21:33.803 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-09-16T19:21:33.804 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-09-16T19:21:34.012 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-09-16T19:21:34.044 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-09-16T19:21:34.251 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-09-16T19:21:34.417 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-09-16T19:21:34.480 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-09-16T19:21:34.668 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-09-16T19:21:34.669 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-09-16T19:21:34.721 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-09-16T19:21:34.723 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-09-16T19:21:34.897 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-09-16T19:21:34.960 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-09-16T19:21:35.127 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-09-16T19:21:35.129 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-09-16T19:21:35.190 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-09-16T19:21:35.192 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-09-16T19:21:35.367 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-09-16T19:21:35.420 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-09-16T19:21:35.607 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-09-16T19:21:35.661 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-09-16T19:21:35.848 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-09-16T19:21:35.860 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:21:35.902 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-09-16T19:21:35.913 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:21:36.090 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-09-16T19:21:36.092 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-09-16T19:21:36.153 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-09-16T19:21:36.155 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-09-16T19:21:36.320 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-09-16T19:21:36.321 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-09-16T19:21:36.331 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:21:36.332 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:21:36.384 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-09-16T19:21:36.386 INFO:teuthology.task.ansible.out:son", "validate_certs": true} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:21:36.407 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:21:36.562 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-09-16T19:21:36.645 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-09-16T19:21:36.801 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-09-16T19:21:36.803 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-09-16T19:21:36.875 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-09-16T19:21:36.877 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-09-16T19:21:37.030 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-09-16T19:21:37.104 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-09-16T19:21:37.260 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-09-16T19:21:37.334 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-09-16T19:21:37.490 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-09-16T19:21:37.564 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-09-16T19:21:37.731 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-09-16T19:21:37.733 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-09-16T19:21:37.805 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-09-16T19:21:37.807 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-09-16T19:21:37.981 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-09-16T19:21:37.983 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-09-16T19:21:38.034 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-09-16T19:21:38.036 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-09-16T19:21:38.221 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-09-16T19:21:38.264 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-09-16T19:21:38.451 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-09-16T19:21:38.494 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-09-16T19:21:38.682 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-09-16T19:21:38.684 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-09-16T19:21:38.725 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-09-16T19:21:38.726 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-09-16T19:21:38.913 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-09-16T19:21:38.956 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-09-16T19:21:39.154 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-09-16T19:21:39.186 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-09-16T19:21:39.384 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-09-16T19:21:39.416 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-09-16T19:21:39.615 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-09-16T19:21:39.647 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-09-16T19:21:39.835 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-09-16T19:21:39.888 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-09-16T19:21:40.075 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-09-16T19:21:40.077 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-09-16T19:21:40.118 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-09-16T19:21:40.119 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-09-16T19:21:40.305 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-09-16T19:21:40.360 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-09-16T19:21:40.539 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-09-16T19:21:40.582 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-09-16T19:21:40.769 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-09-16T19:21:40.812 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-09-16T19:21:41.011 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-09-16T19:21:41.042 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-09-16T19:21:41.239 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-16T19:21:41.272 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-16T19:21:41.469 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-09-16T19:21:41.501 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-09-16T19:21:41.700 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-09-16T19:21:41.742 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-09-16T19:21:41.929 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-09-16T19:21:41.930 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-09-16T19:21:41.974 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-09-16T19:21:41.976 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-09-16T19:21:42.160 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-09-16T19:21:42.162 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-09-16T19:21:42.202 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-09-16T19:21:42.204 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-09-16T19:21:42.390 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-09-16T19:21:42.457 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-09-16T19:21:42.619 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-09-16T19:21:42.682 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-09-16T19:21:42.858 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-09-16T19:21:42.932 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-09-16T19:21:43.091 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-09-16T19:21:43.093 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-16T19:21:43.156 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-09-16T19:21:43.158 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-16T19:21:43.346 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-09-16T19:21:43.348 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-09-16T19:21:43.350 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-09-16T19:21:43.399 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-09-16T19:21:43.401 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-09-16T19:21:43.403 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-09-16T19:21:43.588 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-09-16T19:21:43.632 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-09-16T19:21:43.822 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-09-16T19:21:43.865 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-09-16T19:21:44.054 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-09-16T19:21:44.096 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-09-16T19:21:44.295 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-09-16T19:21:44.328 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-09-16T19:21:44.526 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-09-16T19:21:44.558 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-09-16T19:21:44.759 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-09-16T19:21:44.761 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-09-16T19:21:44.791 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-09-16T19:21:44.792 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-09-16T19:21:44.979 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-09-16T19:21:45.031 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-09-16T19:21:45.221 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-09-16T19:21:45.265 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-09-16T19:21:45.455 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-09-16T19:21:45.456 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-09-16T19:21:45.509 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-09-16T19:21:45.510 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-09-16T19:21:45.691 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-09-16T19:21:45.745 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-09-16T19:21:45.923 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-09-16T19:21:45.977 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-09-16T19:21:46.177 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-09-16T19:21:46.179 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-09-16T19:21:46.210 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-09-16T19:21:46.212 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-09-16T19:21:46.410 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-09-16T19:21:46.442 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-09-16T19:21:46.641 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-09-16T19:21:46.673 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-09-16T19:21:46.882 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-09-16T19:21:46.914 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-09-16T19:21:47.112 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-16T19:21:47.144 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-16T19:21:47.343 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-09-16T19:21:47.376 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-09-16T19:21:47.573 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-09-16T19:21:47.606 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-09-16T19:21:47.803 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-09-16T19:21:47.835 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-09-16T19:21:48.032 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-09-16T19:21:48.074 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-09-16T19:21:48.261 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-09-16T19:21:48.303 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-09-16T19:21:48.480 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-09-16T19:21:48.543 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-09-16T19:21:48.709 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-09-16T19:21:48.762 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-09-16T19:21:48.938 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-09-16T19:21:48.991 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-09-16T19:21:49.167 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-09-16T19:21:49.219 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-09-16T19:21:49.397 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-09-16T19:21:49.460 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-09-16T19:21:49.626 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-09-16T19:21:49.679 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-09-16T19:21:49.857 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-09-16T19:21:49.909 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-09-16T19:21:50.086 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-09-16T19:21:50.139 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-09-16T19:21:50.316 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-09-16T19:21:50.368 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-09-16T19:21:50.545 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-09-16T19:21:50.597 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-09-16T19:21:50.774 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-09-16T19:21:50.827 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-09-16T19:21:51.003 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-09-16T19:21:51.005 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-09-16T19:21:51.066 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-09-16T19:21:51.068 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-09-16T19:21:51.232 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-09-16T19:21:51.295 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-09-16T19:21:51.466 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-09-16T19:21:51.530 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-09-16T19:21:51.706 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-09-16T19:21:51.762 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-09-16T19:21:51.938 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-09-16T19:21:51.940 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-09-16T19:21:51.990 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-09-16T19:21:51.992 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-09-16T19:21:52.166 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-09-16T19:21:52.219 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-09-16T19:21:52.406 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-09-16T19:21:52.459 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-09-16T19:21:52.635 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-09-16T19:21:52.688 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-09-16T19:21:52.875 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-09-16T19:21:52.928 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-09-16T19:21:53.106 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-09-16T19:21:53.159 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-09-16T19:21:53.335 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-09-16T19:21:53.388 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-09-16T19:21:53.565 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-09-16T19:21:53.619 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-09-16T19:21:53.798 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-09-16T19:21:53.800 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-09-16T19:21:53.851 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-09-16T19:21:53.853 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-09-16T19:21:54.027 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-09-16T19:21:54.080 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-09-16T19:21:54.256 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-09-16T19:21:54.309 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-09-16T19:21:54.486 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-09-16T19:21:54.539 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-09-16T19:21:54.707 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-09-16T19:21:54.770 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-09-16T19:21:54.946 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-09-16T19:21:54.998 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-09-16T19:21:55.175 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-09-16T19:21:55.228 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-09-16T19:21:55.404 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-09-16T19:21:55.406 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-16T19:21:55.456 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-09-16T19:21:55.458 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-16T19:21:55.632 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-09-16T19:21:55.685 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-09-16T19:21:55.863 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-09-16T19:21:55.915 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-09-16T19:21:56.092 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-09-16T19:21:56.145 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-09-16T19:21:56.321 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-09-16T19:21:56.374 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-09-16T19:21:56.551 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-09-16T19:21:56.603 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-09-16T19:21:56.780 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-09-16T19:21:56.832 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-09-16T19:21:57.009 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-09-16T19:21:57.071 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-09-16T19:21:57.238 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-09-16T19:21:57.290 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-09-16T19:21:57.467 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-09-16T19:21:57.520 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-09-16T19:21:57.696 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-09-16T19:21:57.759 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-09-16T19:21:57.936 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-09-16T19:21:57.988 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-09-16T19:21:58.155 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-09-16T19:21:58.218 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-09-16T19:21:58.384 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-09-16T19:21:58.447 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-09-16T19:21:58.613 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-09-16T19:21:58.676 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-09-16T19:21:58.852 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-09-16T19:21:58.905 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-09-16T19:21:59.081 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-09-16T19:21:59.134 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-09-16T19:21:59.310 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-09-16T19:21:59.362 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-09-16T19:21:59.538 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-09-16T19:21:59.591 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-09-16T19:21:59.768 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-09-16T19:21:59.770 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-09-16T19:21:59.820 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-09-16T19:21:59.822 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-09-16T19:21:59.996 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-09-16T19:22:00.049 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-09-16T19:22:00.226 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-09-16T19:22:00.288 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-09-16T19:22:00.454 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-09-16T19:22:00.517 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-09-16T19:22:00.683 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-09-16T19:22:00.746 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-09-16T19:22:00.913 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-09-16T19:22:00.976 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-09-16T19:22:01.152 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-09-16T19:22:01.214 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-09-16T19:22:01.380 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-09-16T19:22:01.443 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-09-16T19:22:01.611 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-09-16T19:22:01.674 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-09-16T19:22:01.832 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-09-16T19:22:01.906 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-09-16T19:22:02.135 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-09-16T19:22:02.136 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-09-16T19:22:02.356 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-09-16T19:22:02.358 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-09-16T19:22:02.586 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-09-16T19:22:02.597 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-09-16T19:22:02.824 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-09-16T19:22:02.826 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-09-16T19:22:03.054 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-09-16T19:22:03.065 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-09-16T19:22:03.283 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-09-16T19:22:03.314 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-09-16T19:22:03.512 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-09-16T19:22:03.514 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-09-16T19:22:03.544 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-09-16T19:22:03.545 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-09-16T19:22:03.741 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-09-16T19:22:03.772 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-09-16T19:22:03.969 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-09-16T19:22:04.001 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-09-16T19:22:04.208 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-09-16T19:22:04.229 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-09-16T19:22:04.436 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-09-16T19:22:04.457 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-09-16T19:22:04.665 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-09-16T19:22:04.687 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-09-16T19:22:04.905 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-09-16T19:22:04.916 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-09-16T19:22:05.134 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-09-16T19:22:05.145 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-09-16T19:22:05.363 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-09-16T19:22:05.385 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-09-16T19:22:05.593 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-09-16T19:22:05.614 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-09-16T19:22:05.823 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-09-16T19:22:05.844 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-09-16T19:22:06.062 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-09-16T19:22:06.074 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-09-16T19:22:06.302 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-09-16T19:22:06.313 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-09-16T19:22:06.531 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-09-16T19:22:06.542 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-09-16T19:22:06.771 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-09-16T19:22:06.773 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-09-16T19:22:07.004 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-09-16T19:22:07.006 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-09-16T19:22:07.233 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-09-16T19:22:07.235 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-09-16T19:22:07.462 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-09-16T19:22:07.473 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-09-16T19:22:07.701 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-09-16T19:22:07.703 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-09-16T19:22:07.931 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-09-16T19:22:07.933 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-09-16T19:22:08.159 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-09-16T19:22:08.160 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-09-16T19:22:08.388 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-09-16T19:22:08.389 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-09-16T19:22:08.627 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-09-16T19:22:08.627 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKSdu3rI8o3qId+GU5KYsxB9CLPTODJFiwy+ww6SgrNKgPfiaSUsZAvcQwoxHfuaEXj6VGaOS6HyFFmvHKhzbecSQCzPIucIuKJJDA7VTaA9Jet0aT2b8Vk4p0Ud9j4WoausmxEdBQC+ZpJxSuqajplejYK1Yi3YEqXSCvM6Md76HCustS8efPvzLsLYttDd34Lw0/zp7ALE8hrepLYJgzQS5Z6btvQEjL7lvtHk80/Y+M6ZIQAl6nPcjb8u+wfs/lxcRA5i94GsZhEoYAeMQptJNvOaA5vYK4QtNyV6lyjGVJl4RUUAtLKKp4JE/kdx7sa0Gy99L+u4MjzBoP5B7SkdJnJx7FioXEen5l5DpiJ6xRixw+SjbqEffW4HHI5+1k6jP6kuHPbWfcrw693+aiFVDQA+myzfWo7iDTjY2H5CtnCh+wr1t3VktXhoyPob/XfMPzMNg4W8nA9KVePwzEXSy2Ro0t71GiFQqBviUuqfktqz4a52PqGyFHtskhFAH3H/h6JYM88vfc7Bw7P/u4eVaoCNgNIbLA/kmf689Q1H2KENj2SX7CDwNL9GCOy/pg89qai63+PFT8258zTi0TywWduP1D0DXXtIB3e5K1yK4Zt6VCinXZAze4bsT+z8y4f9W2Xs/FhhtUmo3mOiDJuaGmuWRzHQQTjL//v/Rimw== blaine@brgardne-mac", "key_options": null, "keyfile": "/home/blaine/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "blaine", "validate_certs": true} 2021-09-16T19:22:08.757 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:08 +0000 (0:00:35.231) 0:01:42.175 **** 2021-09-16T19:22:08.778 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:08 +0000 (0:00:00.021) 0:01:42.196 **** 2021-09-16T19:22:08.841 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-09-16T19:22:08.842 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:08.852 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:08.853 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:08.863 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:08.865 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:08.875 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:08.877 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:08.888 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:08.889 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:08.900 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:08.901 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:08.911 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:08.913 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.121 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-09-16T19:22:09.132 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.133 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.144 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-09-16T19:22:09.146 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.156 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.363 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-09-16T19:22:09.385 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-09-16T19:22:09.593 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-09-16T19:22:09.595 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.604 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.615 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-09-16T19:22:09.616 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.617 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.617 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.628 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.630 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.630 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e30 2021-09-16T19:22:09.632 INFO:teuthology.task.ansible.out:7bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.641 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.642 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.653 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.654 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.665 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.666 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.677 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.678 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.689 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.690 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.701 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.703 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.713 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.714 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.725 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.726 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.737 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.738 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.749 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.751 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.761 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.763 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.773 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.774 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.785 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.787 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.798 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.799 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.810 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.812 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.822 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.824 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.834 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.835 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.846 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.847 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.847 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.849 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.858 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.860 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.871 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.873 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.883 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.885 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.896 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.897 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.898 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.909 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.910 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.921 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.922 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.933 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.934 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.944 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.946 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.957 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.959 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.968 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.970 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.981 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.983 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.993 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:09.995 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.005 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.007 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.017 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.018 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.029 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.030 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.041 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.042 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.042 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.043 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.055 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.057 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "k 2021-09-16T19:22:10.058 INFO:teuthology.task.ansible.out:keithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.067 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.068 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.079 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.081 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.091 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.102 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.104 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.114 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.116 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.127 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.128 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.129 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.139 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.142 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.152 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.154 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.164 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.165 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.177 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.188 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.190 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.201 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.202 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.212 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.214 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.224 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.226 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.236 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.238 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.248 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.250 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.260 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.262 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.272 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.274 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.285 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.286 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.296 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.298 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.308 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.310 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.320 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.322 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.333 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.334 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.334 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.335 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.345 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.347 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.357 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.358 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.369 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.371 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.382 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.384 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.394 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.396 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.406 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.408 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.419 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.420 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.430 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.432 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.432 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.432 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": fals 2021-09-16T19:22:10.434 INFO:teuthology.task.ansible.out:e, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.443 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.444 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.455 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.457 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.468 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.469 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.480 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.481 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.492 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.493 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.504 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.504 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.505 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.505 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.516 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.518 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.528 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.529 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.540 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.542 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.561 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.563 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.583 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:10 +0000 (0:00:01.805) 0:01:44.001 **** 2021-09-16T19:22:10.591 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.593 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item={'name': 'blaine', 'ovpn': 'blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "blaine", "ovpn": "blaine@laptop RkDSJBkoRKI0UaxdQuMNCw abcfdade65dd772c066c0c63e712fb03240d676d72eb341186b0c0a94d4a4547"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.637 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:10 +0000 (0:00:00.053) 0:01:44.055 **** 2021-09-16T19:22:10.654 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-09-16T19:22:10.656 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.677 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:10 +0000 (0:00:00.040) 0:01:44.095 **** 2021-09-16T19:22:10.705 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:10.728 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:10 +0000 (0:00:00.051) 0:01:44.146 **** 2021-09-16T19:22:10.953 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-09-16T19:22:10.955 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-09-16T19:22:10.996 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-09-16T19:22:11.142 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-09-16T19:22:11.196 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-09-16T19:22:11.342 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-09-16T19:22:11.395 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-09-16T19:22:11.530 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-09-16T19:22:11.583 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-09-16T19:22:11.719 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-09-16T19:22:11.771 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-09-16T19:22:11.907 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-09-16T19:22:11.960 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-09-16T19:22:12.106 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-09-16T19:22:12.149 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-09-16T19:22:12.295 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-09-16T19:22:12.348 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-09-16T19:22:12.483 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-09-16T19:22:12.536 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-09-16T19:22:12.671 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-09-16T19:22:12.724 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-09-16T19:22:12.859 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-09-16T19:22:12.922 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-09-16T19:22:13.057 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-09-16T19:22:13.109 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-09-16T19:22:13.244 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-09-16T19:22:13.297 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-09-16T19:22:13.433 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-09-16T19:22:13.485 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-09-16T19:22:13.630 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-09-16T19:22:13.672 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-09-16T19:22:13.818 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-09-16T19:22:13.860 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-09-16T19:22:14.007 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-09-16T19:22:14.050 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-09-16T19:22:14.195 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-09-16T19:22:14.248 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-09-16T19:22:14.393 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-09-16T19:22:14.436 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-09-16T19:22:14.582 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-09-16T19:22:14.623 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-09-16T19:22:14.769 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-09-16T19:22:14.812 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-09-16T19:22:14.967 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-09-16T19:22:15.009 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-09-16T19:22:15.155 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-09-16T19:22:15.197 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-09-16T19:22:15.343 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-09-16T19:22:15.385 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-09-16T19:22:15.531 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-09-16T19:22:15.573 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-09-16T19:22:15.730 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-09-16T19:22:15.761 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-09-16T19:22:15.919 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-09-16T19:22:15.951 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-09-16T19:22:16.118 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-09-16T19:22:16.150 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-09-16T19:22:16.305 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-09-16T19:22:16.337 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-09-16T19:22:16.493 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-09-16T19:22:16.525 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-09-16T19:22:16.693 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-09-16T19:22:16.715 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-09-16T19:22:16.883 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-09-16T19:22:16.915 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-09-16T19:22:17.072 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-09-16T19:22:17.114 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-09-16T19:22:17.260 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-09-16T19:22:17.302 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-09-16T19:22:17.459 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-09-16T19:22:17.502 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-09-16T19:22:17.658 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-09-16T19:22:17.690 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-09-16T19:22:17.838 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-09-16T19:22:17.881 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-09-16T19:22:18.038 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-09-16T19:22:18.070 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-09-16T19:22:18.227 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-09-16T19:22:18.269 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-09-16T19:22:19.791 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2021-09-16T19:22:19.834 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item=davidz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "davidz", "name": "davidz", "remove": false, "state": "absent"} 2021-09-16T19:22:21.387 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2021-09-16T19:22:21.419 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item=wusui) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "wusui", "name": "wusui", "remove": false, "state": "absent"} 2021-09-16T19:22:22.952 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2021-09-16T19:22:22.974 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item=nwatkins) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "nwatkins", "name": "nwatkins", "remove": false, "state": "absent"} 2021-09-16T19:22:24.486 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2021-09-16T19:22:24.539 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item=sidharthanup) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sidharthanup", "name": "sidharthanup", "remove": false, "state": "absent"} 2021-09-16T19:22:26.056 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2021-09-16T19:22:26.067 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item=varsha) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "varsha", "name": "varsha", "remove": false, "state": "absent"} 2021-09-16T19:22:27.599 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2021-09-16T19:22:27.610 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item=hmunjulu) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "hmunjulu", "name": "hmunjulu", "remove": false, "state": "absent"} 2021-09-16T19:22:27.800 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-09-16T19:22:27.811 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-09-16T19:22:27.999 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-09-16T19:22:28.021 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-09-16T19:22:28.198 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-09-16T19:22:28.230 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-09-16T19:22:28.398 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-09-16T19:22:28.420 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-09-16T19:22:28.587 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-09-16T19:22:28.608 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-09-16T19:22:28.775 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-09-16T19:22:28.797 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-09-16T19:22:28.964 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-09-16T19:22:28.996 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-09-16T19:22:29.153 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-09-16T19:22:29.185 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-09-16T19:22:29.341 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-09-16T19:22:29.374 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-09-16T19:22:29.531 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-09-16T19:22:29.583 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:29 +0000 (0:00:18.854) 0:02:03.001 **** 2021-09-16T19:22:29.589 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-09-16T19:22:29.636 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:29 +0000 (0:00:00.052) 0:02:03.054 **** 2021-09-16T19:22:30.122 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-09-16T19:22:30.124 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "checksum": "4a5787f42b17837e1416d13c4325647102121755", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "c85c4869d3df1b2c6c21c5a0709ecc0c", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1631820149.6776927-3357-160770833636466/source", "state": "file", "uid": 0} 2021-09-16T19:22:30.145 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:30 +0000 (0:00:00.509) 0:02:03.563 **** 2021-09-16T19:22:30.173 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "checksum": "4a5787f42b17837e1416d13c4325647102121755", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "c85c4869d3df1b2c6c21c5a0709ecc0c", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1631820149.7264936-3362-223649285780812/source", "state": "file", "uid": 0} 2021-09-16T19:22:30.196 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:30 +0000 (0:00:00.050) 0:02:03.614 **** 2021-09-16T19:22:30.212 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-09-16T19:22:30.214 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => { "msg": "Host smithi118.front.sepia.ceph.com is running RedHat 8.4 (Ootpa)" } 2021-09-16T19:22:30.235 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:30 +0000 (0:00:00.039) 0:02:03.653 **** 2021-09-16T19:22:30.263 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => { "msg": "Host smithi109.front.sepia.ceph.com is running RedHat 8.4 (Ootpa)" } 2021-09-16T19:22:30.286 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:30 +0000 (0:00:00.050) 0:02:03.703 **** 2021-09-16T19:22:30.305 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-09-16T19:22:30.307 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"], "rhsm_repos": ["rhel-8-for-x86_64-baseos-rpms", "rhel-8-for-x86_64-appstream-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"} 2021-09-16T19:22:30.329 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:30 +0000 (0:00:00.043) 0:02:03.747 **** 2021-09-16T19:22:30.356 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"], "rhsm_repos": ["rhel-8-for-x86_64-baseos-rpms", "rhel-8-for-x86_64-appstream-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"} 2021-09-16T19:22:30.378 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:30 +0000 (0:00:00.049) 0:02:03.796 **** 2021-09-16T19:22:30.395 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-09-16T19:22:30.397 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:30.418 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:30 +0000 (0:00:00.039) 0:02:03.836 **** 2021-09-16T19:22:30.445 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:30.468 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:30 +0000 (0:00:00.049) 0:02:03.886 **** 2021-09-16T19:22:30.723 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-09-16T19:22:30.725 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.072919", "end": "2021-09-16 19:22:30.701051", "rc": 0, "start": "2021-09-16 19:22:30.628132", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-09-16T19:22:30.746 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:30 +0000 (0:00:00.277) 0:02:04.164 **** 2021-09-16T19:22:30.762 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.065953", "end": "2021-09-16 19:22:30.744455", "rc": 0, "start": "2021-09-16 19:22:30.678502", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-09-16T19:22:30.787 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:30 +0000 (0:00:00.041) 0:02:04.205 **** 2021-09-16T19:22:30.817 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:30.840 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:30 +0000 (0:00:00.053) 0:02:04.258 **** 2021-09-16T19:22:30.858 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:30.881 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:30 +0000 (0:00:00.040) 0:02:04.299 **** 2021-09-16T19:22:30.919 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-09-16T19:22:30.921 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:30.942 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:30 +0000 (0:00:00.060) 0:02:04.360 **** 2021-09-16T19:22:30.959 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:30.981 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:30 +0000 (0:00:00.039) 0:02:04.399 **** 2021-09-16T19:22:31.019 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-09-16T19:22:31.021 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:31.046 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:31 +0000 (0:00:00.064) 0:02:04.464 **** 2021-09-16T19:22:31.064 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:31.088 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:31 +0000 (0:00:00.041) 0:02:04.506 **** 2021-09-16T19:22:31.116 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-09-16T19:22:31.118 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:31.140 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:31 +0000 (0:00:00.052) 0:02:04.558 **** 2021-09-16T19:22:31.158 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:31.180 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:31 +0000 (0:00:00.040) 0:02:04.598 **** 2021-09-16T19:22:31.208 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-09-16T19:22:31.210 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:31.232 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:31 +0000 (0:00:00.051) 0:02:04.649 **** 2021-09-16T19:22:31.249 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:31.272 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:31 +0000 (0:00:00.040) 0:02:04.690 **** 2021-09-16T19:22:31.311 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-09-16T19:22:31.313 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-09-16T19:22:31.335 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:31 +0000 (0:00:00.062) 0:02:04.753 **** 2021-09-16T19:22:31.354 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-09-16T19:22:31.377 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:31 +0000 (0:00:00.041) 0:02:04.795 **** 2021-09-16T19:22:31.418 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** ok: [smithi118.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-16T19:22:31.420 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-16T19:22:31.447 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:31 +0000 (0:00:00.069) 0:02:04.865 **** 2021-09-16T19:22:31.465 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-16T19:22:31.466 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-16T19:22:31.488 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:31 +0000 (0:00:00.041) 0:02:04.906 **** 2021-09-16T19:22:31.527 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-09-16T19:22:31.529 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-09-16T19:22:31.552 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:31 +0000 (0:00:00.063) 0:02:04.970 **** 2021-09-16T19:22:31.581 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-09-16T19:22:31.604 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:31 +0000 (0:00:00.051) 0:02:05.022 **** 2021-09-16T19:22:32.851 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-09-16T19:22:32.853 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed /root/.ansible/tmp/ansible-moduletmp-1631820152.70656-9co2yik9/katello-ca-consumer-latest.noarchpg0noigk.rpm"]} 2021-09-16T19:22:32.873 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:32 +0000 (0:00:01.269) 0:02:06.291 **** 2021-09-16T19:22:32.881 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed /root/.ansible/tmp/ansible-moduletmp-1631820152.7066636-wo8y23_7/katello-ca-consumer-latest.noarchx9zdkhye.rpm"]} 2021-09-16T19:22:32.905 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:32 +0000 (0:00:00.031) 0:02:06.322 **** 2021-09-16T19:22:33.603 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-09-16T19:22:33.605 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "checksum": "532cb5b70a3f71ce180c165205647e8909d4e8a0", "dest": "/etc/yum/vars/releasever", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/yum/vars/releasever", "secontext": "system_u:object_r:etc_t:s0", "size": 3, "state": "file", "uid": 0} ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "532cb5b70a3f71ce180c165205647e8909d4e8a0", "dest": "/etc/yum/vars/releasever", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/yum/vars/releasever", "secontext": "system_u:object_r:etc_t:s0", "size": 3, "state": "file", "uid": 0} 2021-09-16T19:22:33.626 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:33 +0000 (0:00:00.720) 0:02:07.043 **** 2021-09-16T19:22:33.647 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:33 +0000 (0:00:00.021) 0:02:07.065 **** 2021-09-16T19:22:34.287 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** fatal: [smithi109.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-09-16T19:22:34.308 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:34 +0000 (0:00:00.661) 0:02:07.726 **** 2021-09-16T19:22:34.316 INFO:teuthology.task.ansible.out:fatal: [smithi118.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-09-16T19:22:34.341 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:34 +0000 (0:00:00.032) 0:02:07.759 **** 2021-09-16T19:22:34.379 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-09-16T19:22:34.381 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-09-16T19:22:34.403 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:34 +0000 (0:00:00.061) 0:02:07.821 **** 2021-09-16T19:22:34.421 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-09-16T19:22:34.445 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:34 +0000 (0:00:00.042) 0:02:07.863 **** 2021-09-16T19:22:51.670 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-09-16T19:22:51.672 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-16T19:22:51.693 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:51 +0000 (0:00:17.247) 0:02:25.110 **** 2021-09-16T19:22:51.772 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-09-16T19:22:51.774 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-09-16T19:22:51.793 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:51 +0000 (0:00:00.100) 0:02:25.211 **** 2021-09-16T19:22:52.376 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-09-16T19:22:52.378 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-16T19:22:52.398 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:52 +0000 (0:00:00.604) 0:02:25.816 **** 2021-09-16T19:22:52.478 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-09-16T19:22:52.480 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-09-16T19:22:52.501 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:52 +0000 (0:00:00.102) 0:02:25.919 **** 2021-09-16T19:22:52.508 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-09-16T19:22:52.510 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.479334", "end": "2021-09-16 19:22:52.480549", "failed_when_result": false, "rc": 0, "start": "2021-09-16 19:22:52.001215", "stderr": "", "stderr_lines": [], "stdout": "7.8\n7.9\n7Server\n8\n8.2\n8.3\n8.4", "stdout_lines": ["7.8", "7.9", "7Server", "8", "8.2", "8.3", "8.4"]} 2021-09-16T19:22:52.537 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:52 +0000 (0:00:00.036) 0:02:25.955 **** 2021-09-16T19:22:52.618 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-09-16T19:22:52.619 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:52.639 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:52 +0000 (0:00:00.102) 0:02:26.057 **** 2021-09-16T19:22:52.708 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:52.731 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:52 +0000 (0:00:00.091) 0:02:26.149 **** 2021-09-16T19:22:52.799 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-09-16T19:22:52.801 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-16T19:22:52.821 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:52 +0000 (0:00:00.089) 0:02:26.239 **** 2021-09-16T19:22:52.889 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:52.911 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:52 +0000 (0:00:00.089) 0:02:26.329 **** 2021-09-16T19:22:53.189 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-09-16T19:22:53.191 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.433701", "end": "2021-09-16 19:22:53.163290", "failed_when_result": false, "rc": 0, "start": "2021-09-16 19:22:52.729589", "stderr": "", "stderr_lines": [], "stdout": "7.8\n7.9\n7Server\n8\n8.2\n8.3\n8.4", "stdout_lines": ["7.8", "7.9", "7Server", "8", "8.2", "8.3", "8.4"]} 2021-09-16T19:22:53.212 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:53 +0000 (0:00:00.301) 0:02:26.630 **** 2021-09-16T19:22:53.291 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-09-16T19:22:53.293 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:53.313 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:53 +0000 (0:00:00.100) 0:02:26.730 **** 2021-09-16T19:22:53.392 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-09-16T19:22:53.394 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:53.415 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:53 +0000 (0:00:00.102) 0:02:26.833 **** 2021-09-16T19:22:53.484 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-09-16T19:22:53.486 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-16T19:22:53.507 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:53 +0000 (0:00:00.091) 0:02:26.925 **** 2021-09-16T19:22:53.577 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:53.598 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:53 +0000 (0:00:00.091) 0:02:27.016 **** 2021-09-16T19:22:55.820 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:02.678871", "end": "2021-09-16 19:22:55.800362", "rc": 0, "start": "2021-09-16 19:22:53.121491", "stderr": "", "stderr_lines": [], "stdout": "rhel-8-for-x86_64-appstream-rpms\nrhel-8-for-x86_64-baseos-rpms", "stdout_lines": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]} 2021-09-16T19:22:55.843 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:55 +0000 (0:00:02.244) 0:02:29.261 **** 2021-09-16T19:22:55.924 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]}, "changed": false} 2021-09-16T19:22:55.946 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:55 +0000 (0:00:00.102) 0:02:29.364 **** 2021-09-16T19:22:56.015 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-09-16T19:22:56.017 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:56.038 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:56 +0000 (0:00:00.092) 0:02:29.456 **** 2021-09-16T19:22:56.117 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-09-16T19:22:56.119 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": false}, "changed": false} 2021-09-16T19:22:56.139 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:56 +0000 (0:00:00.101) 0:02:29.557 **** 2021-09-16T19:22:56.218 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-09-16T19:22:56.220 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-09-16T19:22:56.247 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:56 +0000 (0:00:00.107) 0:02:29.665 **** 2021-09-16T19:22:56.420 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-09-16T19:22:56.422 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:02.584531", "end": "2021-09-16 19:22:56.400236", "rc": 0, "start": "2021-09-16 19:22:53.815705", "stderr": "", "stderr_lines": [], "stdout": "rhel-8-for-x86_64-appstream-rpms\nrhel-8-for-x86_64-baseos-rpms", "stdout_lines": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]} 2021-09-16T19:22:56.444 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:56 +0000 (0:00:00.196) 0:02:29.861 **** 2021-09-16T19:22:56.543 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-09-16T19:22:56.544 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]}, "changed": false} 2021-09-16T19:22:56.565 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:56 +0000 (0:00:00.121) 0:02:29.983 **** 2021-09-16T19:22:56.644 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:22:56.666 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:56 +0000 (0:00:00.101) 0:02:30.084 **** 2021-09-16T19:22:56.745 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-09-16T19:22:56.747 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": false}, "changed": false} 2021-09-16T19:22:56.769 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:56 +0000 (0:00:00.103) 0:02:30.187 **** 2021-09-16T19:22:56.849 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-09-16T19:22:56.850 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-09-16T19:22:56.871 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:56 +0000 (0:00:00.101) 0:02:30.289 **** 2021-09-16T19:22:58.792 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-09-16T19:22:58.794 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-16T19:22:58.819 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:58 +0000 (0:00:01.948) 0:02:32.237 **** 2021-09-16T19:22:59.099 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-16T19:22:59.122 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:22:59 +0000 (0:00:00.302) 0:02:32.540 **** 2021-09-16T19:23:02.464 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-09-16T19:23:02.466 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-8-for-x86_64-baseos-rpms", "--enable=rhel-8-for-x86_64-appstream-rpms"], "delta": "0:00:03.411070", "end": "2021-09-16 19:23:02.443605", "rc": 0, "start": "2021-09-16 19:22:59.032535", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.\nRepository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.", "Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system."]} 2021-09-16T19:23:02.487 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:02 +0000 (0:00:03.364) 0:02:35.904 **** 2021-09-16T19:23:02.557 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-8-for-x86_64-baseos-rpms", "--enable=rhel-8-for-x86_64-appstream-rpms"], "delta": "0:00:03.201530", "end": "2021-09-16 19:23:02.537147", "rc": 0, "start": "2021-09-16 19:22:59.335617", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.\nRepository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.", "Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system."]} 2021-09-16T19:23:02.585 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:02 +0000 (0:00:00.098) 0:02:36.003 **** 2021-09-16T19:23:02.718 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-09-16T19:23:02.720 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-09-16T19:23:02.746 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:02 +0000 (0:00:00.160) 0:02:36.164 **** 2021-09-16T19:23:02.826 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-09-16T19:23:02.850 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:02 +0000 (0:00:00.104) 0:02:36.268 **** 2021-09-16T19:23:02.981 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-09-16T19:23:02.983 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-16T19:23:03.005 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:03 +0000 (0:00:00.155) 0:02:36.423 **** 2021-09-16T19:23:03.085 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-16T19:23:03.108 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:03 +0000 (0:00:00.102) 0:02:36.526 **** 2021-09-16T19:23:03.533 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-09-16T19:23:03.535 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-09-16T19:23:03.630 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-09-16T19:23:03.963 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-09-16T19:23:03.987 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:03 +0000 (0:00:00.879) 0:02:37.405 **** 2021-09-16T19:23:04.059 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-09-16T19:23:04.099 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:04 +0000 (0:00:00.110) 0:02:37.516 **** 2021-09-16T19:23:04.503 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-09-16T19:23:04.504 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-09-16T19:23:04.598 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-09-16T19:23:04.935 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-09-16T19:23:04.960 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:04 +0000 (0:00:00.861) 0:02:38.377 **** 2021-09-16T19:23:05.031 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.057 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.097) 0:02:38.475 **** 2021-09-16T19:23:05.064 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-09-16T19:23:05.066 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-09-16T19:23:05.091 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.033) 0:02:38.509 **** 2021-09-16T19:23:05.120 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-09-16T19:23:05.122 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.143 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.052) 0:02:38.561 **** 2021-09-16T19:23:05.161 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-09-16T19:23:05.163 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.184 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.040) 0:02:38.602 **** 2021-09-16T19:23:05.212 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-09-16T19:23:05.214 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.235 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.050) 0:02:38.653 **** 2021-09-16T19:23:05.252 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-09-16T19:23:05.255 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.280 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.044) 0:02:38.698 **** 2021-09-16T19:23:05.308 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-09-16T19:23:05.309 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.331 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.051) 0:02:38.749 **** 2021-09-16T19:23:05.349 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-09-16T19:23:05.351 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.372 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.040) 0:02:38.790 **** 2021-09-16T19:23:05.400 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-09-16T19:23:05.402 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.411 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.413 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.436 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.063) 0:02:38.853 **** 2021-09-16T19:23:05.442 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.468 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.031) 0:02:38.885 **** 2021-09-16T19:23:05.506 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-09-16T19:23:05.508 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.534 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.066) 0:02:38.951 **** 2021-09-16T19:23:05.541 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-09-16T19:23:05.543 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.553 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.555 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.578 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.044) 0:02:38.996 **** 2021-09-16T19:23:05.595 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-09-16T19:23:05.597 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.619 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.040) 0:02:39.036 **** 2021-09-16T19:23:05.646 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-09-16T19:23:05.648 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.669 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.050) 0:02:39.087 **** 2021-09-16T19:23:05.686 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-09-16T19:23:05.688 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.709 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.040) 0:02:39.127 **** 2021-09-16T19:23:05.737 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-09-16T19:23:05.739 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.760 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.050) 0:02:39.178 **** 2021-09-16T19:23:05.777 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-09-16T19:23:05.779 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.801 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.041) 0:02:39.219 **** 2021-09-16T19:23:05.829 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-09-16T19:23:05.831 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.853 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.051) 0:02:39.271 **** 2021-09-16T19:23:05.872 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-09-16T19:23:05.875 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.896 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.043) 0:02:39.314 **** 2021-09-16T19:23:05.926 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-09-16T19:23:05.928 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:05.952 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:05 +0000 (0:00:00.056) 0:02:39.370 **** 2021-09-16T19:23:06.024 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-09-16T19:23:06.026 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:06.047 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:06 +0000 (0:00:00.094) 0:02:39.465 **** 2021-09-16T19:23:06.521 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-09-16T19:23:06.523 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.432554", "end": "2021-09-16 19:23:06.500361", "rc": 0, "start": "2021-09-16 19:23:06.067807", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.282422", "end": "2021-09-16 19:23:06.500355", "rc": 0, "start": "2021-09-16 19:23:06.217933", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:23:06.544 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:06 +0000 (0:00:00.496) 0:02:39.962 **** 2021-09-16T19:23:06.567 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:06 +0000 (0:00:00.022) 0:02:39.985 **** 2021-09-16T19:23:20.507 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-09-16T19:23:20.509 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-16T19:23:20.528 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:20 +0000 (0:00:13.961) 0:02:53.946 **** 2021-09-16T19:23:20.597 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-09-16T19:23:20.599 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:20.620 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:20 +0000 (0:00:00.091) 0:02:54.037 **** 2021-09-16T19:23:20.626 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-09-16T19:23:20.628 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-16T19:23:20.652 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:20 +0000 (0:00:00.032) 0:02:54.070 **** 2021-09-16T19:23:20.691 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-09-16T19:23:20.693 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:20.715 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:20 +0000 (0:00:00.062) 0:02:54.133 **** 2021-09-16T19:23:20.721 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-09-16T19:23:20.722 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:20.745 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:20 +0000 (0:00:00.030) 0:02:54.163 **** 2021-09-16T19:23:20.782 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-09-16T19:23:20.784 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:20.806 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:20 +0000 (0:00:00.060) 0:02:54.224 **** 2021-09-16T19:23:20.814 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-09-16T19:23:20.815 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:20.838 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:20 +0000 (0:00:00.032) 0:02:54.256 **** 2021-09-16T19:23:20.877 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-09-16T19:23:20.879 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:20.901 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:20 +0000 (0:00:00.062) 0:02:54.318 **** 2021-09-16T19:23:20.908 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-09-16T19:23:20.910 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:20.933 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:20 +0000 (0:00:00.032) 0:02:54.351 **** 2021-09-16T19:23:21.001 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-09-16T19:23:21.003 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:21.023 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:21 +0000 (0:00:00.090) 0:02:54.441 **** 2021-09-16T19:23:21.375 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-09-16T19:23:21.377 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-09-16T19:23:21.398 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:21 +0000 (0:00:00.374) 0:02:54.816 **** 2021-09-16T19:23:21.516 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-09-16T19:23:21.539 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:21 +0000 (0:00:00.140) 0:02:54.956 **** 2021-09-16T19:23:24.343 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-09-16T19:23:24.345 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-16T19:23:24.366 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:24 +0000 (0:00:02.827) 0:02:57.784 **** 2021-09-16T19:23:24.447 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-16T19:23:24.477 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:24 +0000 (0:00:00.110) 0:02:57.895 **** 2021-09-16T19:23:24.881 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-09-16T19:23:24.932 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-09-16T19:23:24.987 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-09-16T19:23:24.989 INFO:teuthology.task.ansible.out: 2021-09-16T19:23:25.321 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-09-16T19:23:25.426 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-09-16T19:23:25.768 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-09-16T19:23:25.792 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:25 +0000 (0:00:01.315) 0:02:59.210 **** 2021-09-16T19:23:25.851 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-09-16T19:23:25.877 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:25 +0000 (0:00:00.084) 0:02:59.295 **** 2021-09-16T19:23:25.979 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-09-16T19:23:25.981 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-16T19:23:26.004 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:26 +0000 (0:00:00.126) 0:02:59.422 **** 2021-09-16T19:23:26.074 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-16T19:23:26.098 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:26 +0000 (0:00:00.093) 0:02:59.516 **** 2021-09-16T19:23:26.494 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-09-16T19:23:26.496 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-09-16T19:23:26.599 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-09-16T19:23:26.893 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-09-16T19:23:27.019 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-09-16T19:23:27.312 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "state": "file", "uid": 0} 2021-09-16T19:23:27.351 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:27 +0000 (0:00:01.253) 0:03:00.769 **** 2021-09-16T19:23:27.423 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-09-16T19:23:27.425 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:27.450 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:27 +0000 (0:00:00.098) 0:03:00.867 **** 2021-09-16T19:23:27.462 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-09-16T19:23:27.464 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "state": "file", "uid": 0} 2021-09-16T19:23:27.492 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:27 +0000 (0:00:00.042) 0:03:00.910 **** 2021-09-16T19:23:27.531 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-09-16T19:23:27.533 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-09-16T19:23:27.557 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:27 +0000 (0:00:00.064) 0:03:00.975 **** 2021-09-16T19:23:27.564 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-09-16T19:23:27.566 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:27.590 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:27 +0000 (0:00:00.032) 0:03:01.008 **** 2021-09-16T19:23:27.670 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-09-16T19:23:27.693 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:27 +0000 (0:00:00.102) 0:03:01.111 **** 2021-09-16T19:23:27.782 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-09-16T19:23:27.784 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-16T19:23:27.807 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:27 +0000 (0:00:00.114) 0:03:01.225 **** 2021-09-16T19:23:27.899 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-09-16T19:23:27.901 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-16T19:23:27.910 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-16T19:23:27.921 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-09-16T19:23:27.922 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} TASK [common : Install nrpe without epel] ************************************** skipping: [smithi118.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-16T19:23:27.944 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:27 +0000 (0:00:00.136) 0:03:01.362 **** 2021-09-16T19:23:27.951 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-16T19:23:27.976 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:27 +0000 (0:00:00.032) 0:03:01.394 **** 2021-09-16T19:23:28.025 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-16T19:23:28.046 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-16T19:23:28.057 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-16T19:23:28.068 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-16T19:23:28.092 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:28 +0000 (0:00:00.115) 0:03:01.510 **** 2021-09-16T19:23:31.016 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-16T19:23:31.038 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:31 +0000 (0:00:02.946) 0:03:04.456 **** 2021-09-16T19:23:31.067 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-16T19:23:31.090 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:31 +0000 (0:00:00.051) 0:03:04.508 **** 2021-09-16T19:23:31.108 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-09-16T19:23:31.110 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:31.132 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:31 +0000 (0:00:00.041) 0:03:04.550 **** 2021-09-16T19:23:31.182 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:31.206 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:31 +0000 (0:00:00.073) 0:03:04.623 **** 2021-09-16T19:23:31.628 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-09-16T19:23:31.630 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-09-16T19:23:31.652 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:31 +0000 (0:00:00.446) 0:03:05.069 **** 2021-09-16T19:23:31.721 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-09-16T19:23:31.744 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:31 +0000 (0:00:00.092) 0:03:05.162 **** 2021-09-16T19:23:32.152 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-09-16T19:23:32.153 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-09-16T19:23:32.174 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:32 +0000 (0:00:00.429) 0:03:05.592 **** 2021-09-16T19:23:32.226 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-09-16T19:23:32.249 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:32 +0000 (0:00:00.074) 0:03:05.666 **** 2021-09-16T19:23:32.266 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-09-16T19:23:32.268 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-09-16T19:23:32.291 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:32 +0000 (0:00:00.042) 0:03:05.709 **** 2021-09-16T19:23:32.331 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-09-16T19:23:32.343 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-09-16T19:23:32.369 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:32 +0000 (0:00:00.078) 0:03:05.787 **** 2021-09-16T19:23:32.534 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-09-16T19:23:32.536 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-16T19:23:32.558 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:32 +0000 (0:00:00.188) 0:03:05.976 **** 2021-09-16T19:23:32.600 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-16T19:23:32.623 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:32 +0000 (0:00:00.065) 0:03:06.041 **** 2021-09-16T19:23:32.829 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-09-16T19:23:32.831 INFO:teuthology.task.ansible.out:fatal: [smithi118.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-09-16T19:23:32.853 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:32 +0000 (0:00:00.229) 0:03:06.271 **** 2021-09-16T19:23:32.881 INFO:teuthology.task.ansible.out:fatal: [smithi109.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-09-16T19:23:32.905 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:32 +0000 (0:00:00.051) 0:03:06.322 **** 2021-09-16T19:23:32.933 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:32.957 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:32 +0000 (0:00:00.052) 0:03:06.375 **** 2021-09-16T19:23:32.976 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:23:33.001 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:32 +0000 (0:00:00.044) 0:03:06.419 **** 2021-09-16T19:23:33.458 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-09-16T19:23:33.460 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-09-16T19:23:33.492 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:33 +0000 (0:00:00.490) 0:03:06.910 **** 2021-09-16T19:23:33.500 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-09-16T19:23:33.523 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:33 +0000 (0:00:00.031) 0:03:06.941 **** 2021-09-16T19:23:34.157 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-09-16T19:23:34.158 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-09-16 19:18:55 UTC", "ActiveEnterTimestampMonotonic": "22155250", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target systemd-tmpfiles-setup.service systemd-journald.socket network-online.target tmp.mount time-sync.target local-fs.target var-run.mount sysinit.target -.mount system.slice basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-09-16 19:18:55 UTC", "AssertTimestampMonotonic": "22139724", "Before": "getty@tty1.service shutdown.target plymouth-quit.service xdm.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-09-16 19:18:55 UTC", "ConditionT 2021-09-16T19:23:34.159 INFO:teuthology.task.ansible.out:imestampMonotonic": "22139724", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1234", "ExecMainStartTimestamp": "Thu 2021-09-16 19:18:55 UTC", "ExecMainStartTimestampMonotonic": "22154866", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Thu 2021-09-16 19:18:55 UTC] ; stop_time=[n/a] ; pid=1234 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "985", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-09-16 19:18:55 UTC", "InactiveExitTimestampMonotonic": "22155250", "In 2021-09-16T19:23:34.161 INFO:teuthology.task.ansible.out:vocationID": "9f072ecc359247d989341f0230b456f2", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127240", "LimitNPROCSoft": "127240", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127240", "LimitSIGPENDINGSoft": "127240", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1234", "MemoryAccounting": "yes", "MemoryCurrent": "671744", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGr 2021-09-16T19:23:34.163 INFO:teuthology.task.ansible.out:oups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-09-16 19:18:55 UTC", "StateChangeTimestampMonotonic": "22155250", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203584", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "989", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-09-16 19:18:55 UTC", "WatchdogTimestampMonotonic": "22155247", "WatchdogUSec": "0"}} ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "enabled" 2021-09-16T19:23:34.164 INFO:teuthology.task.ansible.out:: true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-09-16 19:19:35 UTC", "ActiveEnterTimestampMonotonic": "22635373", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target systemd-tmpfiles-setup.service -.mount systemd-journald.socket basic.target time-sync.target network-online.target var-run.mount system.slice local-fs.target tmp.mount sysinit.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-09-16 19:19:35 UTC", "AssertTimestampMonotonic": "22620097", "Before": "getty@tty1.service xdm.service plymouth-quit.service shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-09-16 19:19:35 UTC", "ConditionTimestampMonotonic": "22620096", "ConfigurationDirectoryMode": "0755" 2021-09-16T19:23:34.166 INFO:teuthology.task.ansible.out:, "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1249", "ExecMainStartTimestamp": "Thu 2021-09-16 19:19:35 UTC", "ExecMainStartTimestampMonotonic": "22635204", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Thu 2021-09-16 19:19:35 UTC] ; stop_time=[n/a] ; pid=1249 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "985", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-09-16 19:19:35 UTC", "InactiveExitTimestampMonotonic": "22635373", "InvocationID": "eedbf8ffae35433cac64f51c4142ab1a", "JobRunningTimeoutU 2021-09-16T19:23:34.168 INFO:teuthology.task.ansible.out:Sec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127239", "LimitNPROCSoft": "127239", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127239", "LimitSIGPENDINGSoft": "127239", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1249", "MemoryAccounting": "yes", "MemoryCurrent": "663552", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "Pro 2021-09-16T19:23:34.169 INFO:teuthology.task.ansible.out:tectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-09-16 19:19:35 UTC", "StateChangeTimestampMonotonic": "22635373", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203583", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "989", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-09-16 19:19:35 UTC", "WatchdogTimestampMonotonic": "22635371", "WatchdogUSec": "0"}} 2021-09-16T19:23:34.183 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:34 +0000 (0:00:00.659) 0:03:07.601 **** 2021-09-16T19:23:34.205 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:34 +0000 (0:00:00.021) 0:03:07.623 **** 2021-09-16T19:23:34.410 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-09-16T19:23:34.412 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001848", "end": "2021-09-16 19:23:34.394800", "rc": 0, "start": "2021-09-16 19:23:34.392952", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-09-16T19:23:34.434 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:34 +0000 (0:00:00.228) 0:03:07.852 **** 2021-09-16T19:23:34.462 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001820", "end": "2021-09-16 19:23:34.438321", "rc": 0, "start": "2021-09-16 19:23:34.436501", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-09-16T19:23:34.487 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:34 +0000 (0:00:00.052) 0:03:07.905 **** 2021-09-16T19:23:37.482 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-16T19:23:37.504 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:37 +0000 (0:00:03.017) 0:03:10.922 **** 2021-09-16T19:23:37.512 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-16T19:23:37.537 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:23:37 +0000 (0:00:00.032) 0:03:10.955 **** 2021-09-16T19:24:29.840 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-09-16T19:24:29.842 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted-3.14.3-67.el8_4.1.noarch", "Installed: selinux-policy-3.14.3-67.el8_4.1.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch"]} 2021-09-16T19:24:29.862 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:24:29 +0000 (0:00:52.325) 0:04:03.280 **** 2021-09-16T19:24:29.985 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted-3.14.3-67.el8_4.1.noarch", "Installed: selinux-policy-3.14.3-67.el8_4.1.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch"]} 2021-09-16T19:24:30.008 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:24:30 +0000 (0:00:00.145) 0:04:03.426 **** 2021-09-16T19:24:30.598 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-09-16T19:24:30.600 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-09-16T19:24:30.620 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:24:30 +0000 (0:00:00.612) 0:04:04.038 **** 2021-09-16T19:24:30.627 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-09-16T19:24:30.651 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:24:30 +0000 (0:00:00.031) 0:04:04.069 **** 2021-09-16T19:24:46.081 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-09-16T19:24:46.083 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:15.207511", "end": "2021-09-16 19:24:46.063152", "failed_when_result": false, "rc": 0, "start": "2021-09-16 19:24:30.855641", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-09-16T19:24:46.104 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:24:46 +0000 (0:00:15.452) 0:04:19.522 **** 2021-09-16T19:24:46.196 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:15.316604", "end": "2021-09-16 19:24:46.175045", "failed_when_result": false, "rc": 0, "start": "2021-09-16 19:24:30.858441", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-09-16T19:24:46.219 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:24:46 +0000 (0:00:00.114) 0:04:19.637 **** 2021-09-16T19:24:46.604 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-09-16T19:24:46.606 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-09-16T19:24:46.629 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:24:46 +0000 (0:00:00.409) 0:04:20.047 **** 2021-09-16T19:24:46.719 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-09-16T19:24:46.743 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:24:46 +0000 (0:00:00.114) 0:04:20.161 **** 2021-09-16T19:24:46.865 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-09-16T19:24:46.867 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.009702", "end": "2021-09-16 19:24:46.843624", "rc": 0, "start": "2021-09-16 19:24:46.833922", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:24:46.890 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:24:46 +0000 (0:00:00.146) 0:04:20.308 **** 2021-09-16T19:24:46.981 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.011881", "end": "2021-09-16 19:24:46.962130", "rc": 0, "start": "2021-09-16 19:24:46.950249", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:24:47.005 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:24:47 +0000 (0:00:00.114) 0:04:20.423 **** 2021-09-16T19:24:47.125 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-09-16T19:24:47.127 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.013424", "end": "2021-09-16 19:24:47.105195", "rc": 0, "start": "2021-09-16 19:24:47.091771", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:24:47.151 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:24:47 +0000 (0:00:00.145) 0:04:20.568 **** 2021-09-16T19:24:47.241 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.006653", "end": "2021-09-16 19:24:47.215773", "rc": 0, "start": "2021-09-16 19:24:47.209120", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:24:47.265 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:24:47 +0000 (0:00:00.115) 0:04:20.683 **** 2021-09-16T19:25:02.426 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-09-16T19:25:02.428 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.056307", "end": "2021-09-16 19:25:02.407213", "rc": 0, "start": "2021-09-16 19:24:47.350906", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:25:02.454 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:02 +0000 (0:00:15.188) 0:04:35.872 **** 2021-09-16T19:25:02.556 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.069167", "end": "2021-09-16 19:25:02.533950", "rc": 0, "start": "2021-09-16 19:24:47.464783", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:25:02.580 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:02 +0000 (0:00:00.126) 0:04:35.998 **** 2021-09-16T19:25:02.695 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-09-16T19:25:02.696 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-09-16T19:25:02.719 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:02 +0000 (0:00:00.139) 0:04:36.137 **** 2021-09-16T19:25:02.792 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-09-16T19:25:02.794 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-09-16T19:25:02.816 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:02 +0000 (0:00:00.096) 0:04:36.233 **** 2021-09-16T19:25:02.834 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-09-16T19:25:02.836 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-09-16T19:25:02.860 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:02 +0000 (0:00:00.044) 0:04:36.278 **** 2021-09-16T19:25:02.890 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-09-16T19:25:02.892 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-16T19:25:02.924 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:02 +0000 (0:00:00.064) 0:04:36.342 **** 2021-09-16T19:25:02.946 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-09-16T19:25:02.948 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-09-16T19:25:02.983 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:02 +0000 (0:00:00.058) 0:04:36.401 **** 2021-09-16T19:25:03.004 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-09-16T19:25:03.006 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": ["dbench"], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "git-all", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "podman", "redhat-lsb", "firewalld", "wget", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "make", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "autoconf", "gdb", "iozone", "lvm2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"} 2021-09-16T19:25:03.029 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:03 +0000 (0:00:00.045) 0:04:36.446 **** 2021-09-16T19:25:03.057 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-09-16T19:25:03.060 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-16T19:25:03.085 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:03 +0000 (0:00:00.055) 0:04:36.502 **** 2021-09-16T19:25:03.102 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-09-16T19:25:03.104 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-16T19:25:03.134 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:03 +0000 (0:00:00.047) 0:04:36.550 **** 2021-09-16T19:25:03.162 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-09-16T19:25:03.164 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": ["dbench"], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "git-all", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "podman", "redhat-lsb", "firewalld", "wget", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "make", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "autoconf", "gdb", "iozone", "lvm2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"} 2021-09-16T19:25:03.187 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:03 +0000 (0:00:00.055) 0:04:36.605 **** 2021-09-16T19:25:03.267 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-09-16T19:25:03.269 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-16T19:25:03.290 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:03 +0000 (0:00:00.102) 0:04:36.708 **** 2021-09-16T19:25:03.318 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-16T19:25:03.341 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:03 +0000 (0:00:00.050) 0:04:36.758 **** 2021-09-16T19:25:03.483 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-16T19:25:03.505 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:03 +0000 (0:00:00.164) 0:04:36.923 **** 2021-09-16T19:25:03.587 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-09-16T19:25:03.589 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-09-16T19:25:03.615 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:03 +0000 (0:00:00.109) 0:04:37.033 **** 2021-09-16T19:25:03.685 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-09-16T19:25:03.686 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:03.707 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:03 +0000 (0:00:00.091) 0:04:37.125 **** 2021-09-16T19:25:03.745 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-09-16T19:25:03.747 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-09-16T19:25:03.772 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:03 +0000 (0:00:00.064) 0:04:37.189 **** 2021-09-16T19:25:03.780 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:03.803 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:03 +0000 (0:00:00.031) 0:04:37.221 **** 2021-09-16T19:25:03.831 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-09-16T19:25:03.833 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:03.857 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:03 +0000 (0:00:00.053) 0:04:37.275 **** 2021-09-16T19:25:03.928 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-09-16T19:25:03.930 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:03.950 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:03 +0000 (0:00:00.092) 0:04:37.368 **** 2021-09-16T19:25:04.060 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-09-16T19:25:04.062 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-09-16T19:25:04.084 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:04 +0000 (0:00:00.133) 0:04:37.501 **** 2021-09-16T19:25:04.204 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-09-16T19:25:04.206 INFO:teuthology.task.ansible.out: 2021-09-16T19:25:04.229 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:04 +0000 (0:00:00.145) 0:04:37.647 **** 2021-09-16T19:25:04.644 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-09-16T19:25:04.646 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-09-16T19:25:04.668 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:04 +0000 (0:00:00.439) 0:04:38.086 **** 2021-09-16T19:25:04.739 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-09-16T19:25:04.740 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi109"}, "changed": false} 2021-09-16T19:25:04.761 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:04 +0000 (0:00:00.092) 0:04:38.179 **** 2021-09-16T19:25:04.799 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-09-16T19:25:04.801 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-09-16T19:25:04.823 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:04 +0000 (0:00:00.062) 0:04:38.241 **** 2021-09-16T19:25:04.904 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-09-16T19:25:04.906 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi118"}, "changed": false} 2021-09-16T19:25:04.926 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:04 +0000 (0:00:00.103) 0:04:38.344 **** 2021-09-16T19:25:05.561 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-09-16T19:25:05.563 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi109", "ansible_hostname": "smithi109", "ansible_nodename": "smithi109"}, "changed": false, "name": "smithi109"} 2021-09-16T19:25:05.585 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:05 +0000 (0:00:00.658) 0:04:39.002 **** 2021-09-16T19:25:05.594 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi118", "ansible_hostname": "smithi118", "ansible_nodename": "smithi118"}, "changed": false, "name": "smithi118"} 2021-09-16T19:25:05.622 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:05 +0000 (0:00:00.037) 0:04:39.040 **** 2021-09-16T19:25:06.080 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-09-16T19:25:06.082 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-09-16T19:25:06.108 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:06 +0000 (0:00:00.485) 0:04:39.526 **** 2021-09-16T19:25:06.122 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-09-16T19:25:06.158 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:06 +0000 (0:00:00.049) 0:04:39.575 **** 2021-09-16T19:25:06.611 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-09-16T19:25:06.613 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-09-16T19:25:06.643 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:06 +0000 (0:00:00.485) 0:04:40.061 **** 2021-09-16T19:25:06.696 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-09-16T19:25:06.726 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:06 +0000 (0:00:00.082) 0:04:40.144 **** 2021-09-16T19:25:07.240 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-09-16T19:25:07.242 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-09-16T19:25:07.263 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:07 +0000 (0:00:00.536) 0:04:40.681 **** 2021-09-16T19:25:07.333 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-09-16T19:25:07.356 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:07 +0000 (0:00:00.092) 0:04:40.774 **** 2021-09-16T19:25:07.767 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-09-16T19:25:07.769 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-09-16T19:25:07.789 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:07 +0000 (0:00:00.433) 0:04:41.207 **** 2021-09-16T19:25:07.858 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-09-16T19:25:07.881 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:07 +0000 (0:00:00.092) 0:04:41.299 **** 2021-09-16T19:25:08.017 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-09-16T19:25:08.018 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-16T19:25:08.042 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:08 +0000 (0:00:00.160) 0:04:41.460 **** 2021-09-16T19:25:08.113 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-16T19:25:08.149 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:08 +0000 (0:00:00.106) 0:04:41.566 **** 2021-09-16T19:25:08.274 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-09-16T19:25:08.276 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-09-16T19:25:08.297 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:08 +0000 (0:00:00.148) 0:04:41.715 **** 2021-09-16T19:25:08.377 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-09-16T19:25:08.403 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:08 +0000 (0:00:00.105) 0:04:41.821 **** 2021-09-16T19:25:08.559 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-09-16T19:25:08.561 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-09-16T19:25:08.583 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:08 +0000 (0:00:00.179) 0:04:42.000 **** 2021-09-16T19:25:08.664 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-09-16T19:25:08.689 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:08 +0000 (0:00:00.105) 0:04:42.106 **** 2021-09-16T19:25:09.082 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-09-16T19:25:09.083 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-09-16T19:25:09.104 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:09 +0000 (0:00:00.416) 0:04:42.522 **** 2021-09-16T19:25:09.195 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-09-16T19:25:09.218 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:09 +0000 (0:00:00.113) 0:04:42.636 **** 2021-09-16T19:25:09.601 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-09-16T19:25:09.603 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-09-16T19:25:09.624 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:09 +0000 (0:00:00.405) 0:04:43.042 **** 2021-09-16T19:25:09.716 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-09-16T19:25:09.740 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:09 +0000 (0:00:00.115) 0:04:43.158 **** 2021-09-16T19:25:12.040 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-09-16T19:25:12.042 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.182572", "end": "2021-09-16 19:25:12.015276", "rc": 0, "start": "2021-09-16 19:25:09.832704", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:25:12.063 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:12 +0000 (0:00:02.323) 0:04:45.481 **** 2021-09-16T19:25:12.070 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.094297", "end": "2021-09-16 19:25:12.043084", "rc": 0, "start": "2021-09-16 19:25:09.948787", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:25:12.095 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:12 +0000 (0:00:00.031) 0:04:45.513 **** 2021-09-16T19:25:12.133 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-09-16T19:25:12.135 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:12.157 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:12 +0000 (0:00:00.061) 0:04:45.574 **** 2021-09-16T19:25:12.175 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:12.198 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:12 +0000 (0:00:00.042) 0:04:45.616 **** 2021-09-16T19:25:12.237 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-16T19:25:12.239 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:12.260 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:12 +0000 (0:00:00.061) 0:04:45.678 **** 2021-09-16T19:25:12.278 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:12.303 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:12 +0000 (0:00:00.042) 0:04:45.720 **** 2021-09-16T19:25:12.342 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-09-16T19:25:12.344 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-09-16T19:25:12.366 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:12 +0000 (0:00:00.063) 0:04:45.784 **** 2021-09-16T19:25:12.384 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-09-16T19:25:12.409 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:12 +0000 (0:00:00.042) 0:04:45.826 **** 2021-09-16T19:25:12.436 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-09-16T19:25:12.438 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:12.459 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:12 +0000 (0:00:00.050) 0:04:45.877 **** 2021-09-16T19:25:12.477 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:12.500 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:12 +0000 (0:00:00.040) 0:04:45.918 **** 2021-09-16T19:25:12.978 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-09-16T19:25:12.980 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-09-16T19:25:13.012 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:13 +0000 (0:00:00.511) 0:04:46.430 **** 2021-09-16T19:25:13.032 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-09-16T19:25:13.058 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:13 +0000 (0:00:00.046) 0:04:46.476 **** 2021-09-16T19:25:13.087 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-09-16T19:25:13.110 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:13 +0000 (0:00:00.052) 0:04:46.528 **** 2021-09-16T19:25:13.152 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:13 +0000 (0:00:00.041) 0:04:46.570 **** 2021-09-16T19:25:14.015 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-09-16T19:25:14.017 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.671982", "end": "2021-09-16 19:25:13.994919", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-09-16 19:25:13.322937", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Updating Subscription Management repositories.\nRepository successfully enabled.", "stdout_lines": ["Updating Subscription Management repositories.", "Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use 2021-09-16T19:25:14.019 INFO:teuthology.task.ansible.out: command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2021-09-16T19:25:14.040 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:14 +0000 (0:00:00.887) 0:04:47.457 **** 2021-09-16T19:25:14.058 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.653146", "end": "2021-09-16 19:25:14.022324", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-09-16 19:25:13.369178", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Updating Subscription Management repositories.\nRepository successfully enabled.", "stdout_lines": ["Updating Subscription Management repositories.", "Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2021-09-16T19:25:14.083 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:14 +0000 (0:00:00.043) 0:04:47.501 **** 2021-09-16T19:25:14.111 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-09-16T19:25:14.113 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:14.135 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:14 +0000 (0:00:00.051) 0:04:47.553 **** 2021-09-16T19:25:14.155 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:14.179 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:14 +0000 (0:00:00.044) 0:04:47.597 **** 2021-09-16T19:25:14.218 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi109.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-09-16T19:25:14.220 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-09-16T19:25:14.243 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:14 +0000 (0:00:00.063) 0:04:47.660 **** 2021-09-16T19:25:14.260 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-09-16T19:25:14.262 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-09-16T19:25:14.285 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:14 +0000 (0:00:00.042) 0:04:47.703 **** 2021-09-16T19:25:14.316 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-09-16T19:25:14.322 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:14.343 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:14 +0000 (0:00:00.057) 0:04:47.761 **** 2021-09-16T19:25:14.361 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:14.387 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:14 +0000 (0:00:00.043) 0:04:47.805 **** 2021-09-16T19:25:14.415 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-16T19:25:14.417 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:14.438 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:14 +0000 (0:00:00.051) 0:04:47.856 **** 2021-09-16T19:25:14.456 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:14.482 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:14 +0000 (0:00:00.043) 0:04:47.899 **** 2021-09-16T19:25:14.511 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-09-16T19:25:14.512 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:14.546 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:14 +0000 (0:00:00.063) 0:04:47.963 **** 2021-09-16T19:25:14.567 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:14.602 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:14 +0000 (0:00:00.056) 0:04:48.020 **** 2021-09-16T19:25:14.623 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-09-16T19:25:14.625 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:14.650 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:14 +0000 (0:00:00.047) 0:04:48.067 **** 2021-09-16T19:25:14.678 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:14.702 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:14 +0000 (0:00:00.052) 0:04:48.120 **** 2021-09-16T19:25:14.896 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-09-16T19:25:14.897 INFO:teuthology.task.ansible.out:fatal: [smithi109.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007018", "end": "2021-09-16 19:25:14.875422", "msg": "non-zero return code", "rc": 1, "start": "2021-09-16 19:25:14.868404", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-09-16T19:25:14.928 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:14 +0000 (0:00:00.225) 0:04:48.346 **** 2021-09-16T19:25:14.949 INFO:teuthology.task.ansible.out:fatal: [smithi118.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006977", "end": "2021-09-16 19:25:14.929263", "msg": "non-zero return code", "rc": 1, "start": "2021-09-16 19:25:14.922286", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-09-16T19:25:14.977 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:14 +0000 (0:00:00.048) 0:04:48.394 **** 2021-09-16T19:25:15.004 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-09-16T19:25:15.006 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:15.028 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:15 +0000 (0:00:00.051) 0:04:48.446 **** 2021-09-16T19:25:15.046 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:25:15.071 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:15 +0000 (0:00:00.042) 0:04:48.489 **** 2021-09-16T19:25:45.046 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-09-16T19:25:45.048 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64"]} 2021-09-16T19:25:45.069 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:45 +0000 (0:00:29.997) 0:05:18.487 **** 2021-09-16T19:25:45.201 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64"]} 2021-09-16T19:25:45.225 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:25:45 +0000 (0:00:00.156) 0:05:18.643 **** 2021-09-16T19:26:12.452 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-09-16T19:26:12.454 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-09-16T19:26:12.476 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:26:12 +0000 (0:00:27.251) 0:05:45.894 **** 2021-09-16T19:26:12.568 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-09-16T19:26:12.593 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:26:12 +0000 (0:00:00.116) 0:05:46.011 **** 2021-09-16T19:26:44.393 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-16T19:26:44.396 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: qemu-kvm-common-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: qemu-kvm-core-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: qemu-img-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64"]} 2021-09-16T19:26:44.415 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:26:44 +0000 (0:00:31.822) 0:06:17.833 **** 2021-09-16T19:26:44.588 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: qemu-kvm-common-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: qemu-kvm-core-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: qemu-img-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64"]} 2021-09-16T19:26:44.613 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:26:44 +0000 (0:00:00.197) 0:06:18.030 **** 2021-09-16T19:26:58.067 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-09-16T19:26:58.069 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-16T19:26:58.090 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:26:58 +0000 (0:00:13.478) 0:06:31.508 **** 2021-09-16T19:26:58.381 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-16T19:26:58.404 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:26:58 +0000 (0:00:00.313) 0:06:31.822 **** 2021-09-16T19:27:01.090 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-09-16T19:27:01.092 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-16T19:27:01.113 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:01 +0000 (0:00:02.709) 0:06:34.531 **** 2021-09-16T19:27:01.184 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-16T19:27:01.185 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:01.206 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:01 +0000 (0:00:00.092) 0:06:34.623 **** 2021-09-16T19:27:01.507 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-09-16T19:27:01.509 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-16T19:27:01.530 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:01 +0000 (0:00:00.324) 0:06:34.948 **** 2021-09-16T19:27:01.548 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-09-16T19:27:01.550 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "sysinit.target polkit.service system.slice basic.target dbus.socket dbus.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ebtables.service ipset.service ip6tables.service nftables.service shutdown.target iptables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", 2021-09-16T19:27:01.551 INFO:teuthology.task.ansible.out:"Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": " 2021-09-16T19:27:01.553 INFO:teuthology.task.ansible.out:0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127240", "LimitNPROCSoft": "127240", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127240", "LimitSIGPENDINGSoft": "127240", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket sysinit.target system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null 2021-09-16T19:27:01.554 INFO:teuthology.task.ansible.out:", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203584", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-09-16T19:27:01.577 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:01 +0000 (0:00:00.045) 0:06:34.994 **** 2021-09-16T19:27:01.604 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-16T19:27:01.606 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:01.630 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:01 +0000 (0:00:00.053) 0:06:35.048 **** 2021-09-16T19:27:01.903 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ fatal: [smithi109.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-09-16T19:27:01.926 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:01 +0000 (0:00:00.295) 0:06:35.343 **** 2021-09-16T19:27:01.987 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-09-16T19:27:01.988 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.service polkit.service basic.target sysinit.target dbus.socket system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ipset.service nftables.service shutdown.target iptables.service ip6tables.service ebtables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", 2021-09-16T19:27:01.990 INFO:teuthology.task.ansible.out:"Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": " 2021-09-16T19:27:01.992 INFO:teuthology.task.ansible.out:0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127239", "LimitNPROCSoft": "127239", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127239", "LimitSIGPENDINGSoft": "127239", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null 2021-09-16T19:27:01.993 INFO:teuthology.task.ansible.out:", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203583", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-09-16T19:27:02.012 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:02 +0000 (0:00:00.086) 0:06:35.430 **** 2021-09-16T19:27:02.329 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-09-16T19:27:02.330 INFO:teuthology.task.ansible.out:fatal: [smithi118.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-09-16T19:27:02.351 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:02 +0000 (0:00:00.338) 0:06:35.769 **** 2021-09-16T19:27:02.567 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-09-16T19:27:02.569 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-09-16T19:27:02.590 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:02 +0000 (0:00:00.239) 0:06:36.008 **** 2021-09-16T19:27:02.660 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-09-16T19:27:02.662 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Enable SELinux] *********************************************** ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-09-16T19:27:02.683 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:02 +0000 (0:00:00.092) 0:06:36.101 **** 2021-09-16T19:27:02.704 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:02 +0000 (0:00:00.020) 0:06:36.122 **** 2021-09-16T19:27:02.763 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-09-16T19:27:02.765 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:02.785 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:02 +0000 (0:00:00.081) 0:06:36.203 **** 2021-09-16T19:27:02.793 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-09-16T19:27:02.795 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:02.818 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:02 +0000 (0:00:00.031) 0:06:36.235 **** 2021-09-16T19:27:02.855 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-09-16T19:27:02.857 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:02.880 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:02 +0000 (0:00:00.062) 0:06:36.298 **** 2021-09-16T19:27:02.899 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-09-16T19:27:02.901 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:02.925 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:02 +0000 (0:00:00.044) 0:06:36.342 **** 2021-09-16T19:27:02.954 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-09-16T19:27:02.956 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:02.980 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:02 +0000 (0:00:00.055) 0:06:36.398 **** 2021-09-16T19:27:03.009 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-09-16T19:27:03.011 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:03.034 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.053) 0:06:36.452 **** 2021-09-16T19:27:03.052 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-09-16T19:27:03.054 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:03.076 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.042) 0:06:36.494 **** 2021-09-16T19:27:03.105 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-09-16T19:27:03.107 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:03.129 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.052) 0:06:36.546 **** 2021-09-16T19:27:03.146 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-09-16T19:27:03.148 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:03.176 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.047) 0:06:36.594 **** 2021-09-16T19:27:03.206 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-09-16T19:27:03.208 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:03.231 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.055) 0:06:36.649 **** 2021-09-16T19:27:03.259 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-09-16T19:27:03.285 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.053) 0:06:36.703 **** 2021-09-16T19:27:03.303 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-09-16T19:27:03.304 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:03.328 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.042) 0:06:36.746 **** 2021-09-16T19:27:03.346 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-09-16T19:27:03.348 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:03.371 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.043) 0:06:36.789 **** 2021-09-16T19:27:03.390 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-09-16T19:27:03.414 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.042) 0:06:36.832 **** 2021-09-16T19:27:03.442 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-09-16T19:27:03.444 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:03.466 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.052) 0:06:36.884 **** 2021-09-16T19:27:03.487 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-09-16T19:27:03.489 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:03.511 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.045) 0:06:36.929 **** 2021-09-16T19:27:03.550 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-09-16T19:27:03.552 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:03.561 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:03.584 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.072) 0:06:37.002 **** 2021-09-16T19:27:03.591 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-09-16T19:27:03.592 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:03.616 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.031) 0:06:37.033 **** 2021-09-16T19:27:03.654 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-09-16T19:27:03.656 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:03.678 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.062) 0:06:37.096 **** 2021-09-16T19:27:03.697 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-09-16T19:27:03.698 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:03.723 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.044) 0:06:37.141 **** 2021-09-16T19:27:03.752 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-09-16T19:27:03.776 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.052) 0:06:37.193 **** 2021-09-16T19:27:03.794 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-09-16T19:27:03.796 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:03.819 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.043) 0:06:37.237 **** 2021-09-16T19:27:03.848 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:03.872 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.052) 0:06:37.290 **** 2021-09-16T19:27:03.891 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-09-16T19:27:03.914 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.042) 0:06:37.332 **** 2021-09-16T19:27:03.942 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-09-16T19:27:03.944 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:03.968 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:03 +0000 (0:00:00.053) 0:06:37.386 **** 2021-09-16T19:27:03.986 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-09-16T19:27:03.988 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.014 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.045) 0:06:37.432 **** 2021-09-16T19:27:04.042 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-09-16T19:27:04.044 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.066 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.051) 0:06:37.484 **** 2021-09-16T19:27:04.094 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-09-16T19:27:04.096 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.118 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.051) 0:06:37.536 **** 2021-09-16T19:27:04.135 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-16T19:27:04.137 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.159 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.041) 0:06:37.577 **** 2021-09-16T19:27:04.188 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-09-16T19:27:04.189 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.214 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.054) 0:06:37.632 **** 2021-09-16T19:27:04.233 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.258 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.043) 0:06:37.675 **** 2021-09-16T19:27:04.287 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-16T19:27:04.289 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.311 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.054) 0:06:37.729 **** 2021-09-16T19:27:04.328 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-16T19:27:04.329 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.353 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.041) 0:06:37.771 **** 2021-09-16T19:27:04.382 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-09-16T19:27:04.384 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.406 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.052) 0:06:37.823 **** 2021-09-16T19:27:04.423 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-09-16T19:27:04.425 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.451 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.045) 0:06:37.868 **** 2021-09-16T19:27:04.481 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-16T19:27:04.482 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.505 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.054) 0:06:37.923 **** 2021-09-16T19:27:04.523 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-09-16T19:27:04.525 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.548 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.042) 0:06:37.965 **** 2021-09-16T19:27:04.577 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-09-16T19:27:04.579 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.601 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.053) 0:06:38.019 **** 2021-09-16T19:27:04.619 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-09-16T19:27:04.620 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.644 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.042) 0:06:38.062 **** 2021-09-16T19:27:04.672 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-09-16T19:27:04.674 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.696 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.052) 0:06:38.114 **** 2021-09-16T19:27:04.714 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-09-16T19:27:04.716 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.738 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.041) 0:06:38.156 **** 2021-09-16T19:27:04.768 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-09-16T19:27:04.770 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.791 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.052) 0:06:38.209 **** 2021-09-16T19:27:04.808 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-09-16T19:27:04.810 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.834 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.042) 0:06:38.252 **** 2021-09-16T19:27:04.852 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-09-16T19:27:04.854 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.875 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.041) 0:06:38.293 **** 2021-09-16T19:27:04.903 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-09-16T19:27:04.905 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.927 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.051) 0:06:38.344 **** 2021-09-16T19:27:04.945 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:04.968 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:04 +0000 (0:00:00.041) 0:06:38.386 **** 2021-09-16T19:27:04.995 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-09-16T19:27:04.997 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.021 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.052) 0:06:38.439 **** 2021-09-16T19:27:05.040 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.064 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.042) 0:06:38.482 **** 2021-09-16T19:27:05.092 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-09-16T19:27:05.094 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.119 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.053) 0:06:38.536 **** 2021-09-16T19:27:05.137 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-09-16T19:27:05.139 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.162 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.043) 0:06:38.579 **** 2021-09-16T19:27:05.190 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-09-16T19:27:05.192 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.215 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.053) 0:06:38.633 **** 2021-09-16T19:27:05.233 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-09-16T19:27:05.235 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.257 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.041) 0:06:38.675 **** 2021-09-16T19:27:05.284 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-09-16T19:27:05.286 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.308 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.051) 0:06:38.726 **** 2021-09-16T19:27:05.326 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-09-16T19:27:05.327 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.349 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.040) 0:06:38.767 **** 2021-09-16T19:27:05.376 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-09-16T19:27:05.377 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.399 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.050) 0:06:38.817 **** 2021-09-16T19:27:05.416 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-09-16T19:27:05.418 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.440 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.040) 0:06:38.857 **** 2021-09-16T19:27:05.467 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-09-16T19:27:05.469 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.490 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.050) 0:06:38.908 **** 2021-09-16T19:27:05.507 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-09-16T19:27:05.509 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.532 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.041) 0:06:38.950 **** 2021-09-16T19:27:05.571 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.597 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.064) 0:06:39.015 **** 2021-09-16T19:27:05.614 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.638 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.040) 0:06:39.056 **** 2021-09-16T19:27:05.666 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-09-16T19:27:05.668 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.690 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.051) 0:06:39.108 **** 2021-09-16T19:27:05.709 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-09-16T19:27:05.710 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.732 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.042) 0:06:39.150 **** 2021-09-16T19:27:05.760 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.785 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.052) 0:06:39.202 **** 2021-09-16T19:27:05.803 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-09-16T19:27:05.805 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.828 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.043) 0:06:39.246 **** 2021-09-16T19:27:05.856 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.880 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.051) 0:06:39.298 **** 2021-09-16T19:27:05.909 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.933 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.053) 0:06:39.351 **** 2021-09-16T19:27:05.950 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-09-16T19:27:05.952 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:05.976 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:05 +0000 (0:00:00.042) 0:06:39.394 **** 2021-09-16T19:27:06.004 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-09-16T19:27:06.006 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.027 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.051) 0:06:39.445 **** 2021-09-16T19:27:06.044 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-16T19:27:06.046 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.068 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.040) 0:06:39.486 **** 2021-09-16T19:27:06.096 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-09-16T19:27:06.098 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.120 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.051) 0:06:39.538 **** 2021-09-16T19:27:06.136 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-09-16T19:27:06.138 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.160 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.040) 0:06:39.578 **** 2021-09-16T19:27:06.190 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.213 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.052) 0:06:39.631 **** 2021-09-16T19:27:06.230 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-16T19:27:06.232 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.255 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.042) 0:06:39.673 **** 2021-09-16T19:27:06.284 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-09-16T19:27:06.285 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.308 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.052) 0:06:39.725 **** 2021-09-16T19:27:06.326 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.349 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.041) 0:06:39.767 **** 2021-09-16T19:27:06.378 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-16T19:27:06.380 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.402 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.052) 0:06:39.820 **** 2021-09-16T19:27:06.419 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-09-16T19:27:06.421 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.444 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.041) 0:06:39.861 **** 2021-09-16T19:27:06.472 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-16T19:27:06.473 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.495 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.051) 0:06:39.913 **** 2021-09-16T19:27:06.512 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-16T19:27:06.514 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.537 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.042) 0:06:39.955 **** 2021-09-16T19:27:06.565 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-09-16T19:27:06.567 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.588 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.051) 0:06:40.006 **** 2021-09-16T19:27:06.605 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-16T19:27:06.606 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.629 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.040) 0:06:40.047 **** 2021-09-16T19:27:06.657 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-16T19:27:06.659 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.683 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.053) 0:06:40.101 **** 2021-09-16T19:27:06.700 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-16T19:27:06.701 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.727 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.043) 0:06:40.145 **** 2021-09-16T19:27:06.755 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-16T19:27:06.757 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.779 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.051) 0:06:40.196 **** 2021-09-16T19:27:06.796 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.819 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.040) 0:06:40.237 **** 2021-09-16T19:27:06.848 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.872 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.052) 0:06:40.290 **** 2021-09-16T19:27:06.899 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-16T19:27:06.901 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.924 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.052) 0:06:40.342 **** 2021-09-16T19:27:06.942 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-09-16T19:27:06.944 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:06.967 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:06 +0000 (0:00:00.042) 0:06:40.385 **** 2021-09-16T19:27:06.995 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-09-16T19:27:06.997 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.020 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.053) 0:06:40.438 **** 2021-09-16T19:27:07.038 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-16T19:27:07.040 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.062 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.042) 0:06:40.480 **** 2021-09-16T19:27:07.090 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-16T19:27:07.092 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.114 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.051) 0:06:40.532 **** 2021-09-16T19:27:07.132 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-09-16T19:27:07.133 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.155 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.041) 0:06:40.573 **** 2021-09-16T19:27:07.183 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-16T19:27:07.184 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.207 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.051) 0:06:40.624 **** 2021-09-16T19:27:07.224 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-16T19:27:07.226 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.248 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.041) 0:06:40.666 **** 2021-09-16T19:27:07.276 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-16T19:27:07.277 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.299 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.051) 0:06:40.717 **** 2021-09-16T19:27:07.318 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-16T19:27:07.320 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.342 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.042) 0:06:40.760 **** 2021-09-16T19:27:07.371 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-16T19:27:07.372 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.396 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.053) 0:06:40.813 **** 2021-09-16T19:27:07.414 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-16T19:27:07.416 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.441 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.045) 0:06:40.859 **** 2021-09-16T19:27:07.469 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-16T19:27:07.471 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.493 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.051) 0:06:40.910 **** 2021-09-16T19:27:07.514 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.539 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.045) 0:06:40.956 **** 2021-09-16T19:27:07.568 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-16T19:27:07.569 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.592 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.052) 0:06:41.009 **** 2021-09-16T19:27:07.609 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-16T19:27:07.611 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.633 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.041) 0:06:41.051 **** 2021-09-16T19:27:07.660 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-09-16T19:27:07.662 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.685 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.051) 0:06:41.103 **** 2021-09-16T19:27:07.703 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-16T19:27:07.705 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.727 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.041) 0:06:41.145 **** 2021-09-16T19:27:07.754 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-09-16T19:27:07.755 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.778 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.051) 0:06:41.196 **** 2021-09-16T19:27:07.806 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-09-16T19:27:07.808 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.828 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.050) 0:06:41.246 **** 2021-09-16T19:27:07.845 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-09-16T19:27:07.847 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.873 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.044) 0:06:41.291 **** 2021-09-16T19:27:07.900 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-09-16T19:27:07.902 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.923 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.050) 0:06:41.341 **** 2021-09-16T19:27:07.940 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-16T19:27:07.942 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:07.964 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:07 +0000 (0:00:00.040) 0:06:41.382 **** 2021-09-16T19:27:07.993 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-09-16T19:27:07.994 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.017 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.052) 0:06:41.435 **** 2021-09-16T19:27:08.034 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-09-16T19:27:08.036 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.058 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.041) 0:06:41.476 **** 2021-09-16T19:27:08.086 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-16T19:27:08.088 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.110 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.051) 0:06:41.527 **** 2021-09-16T19:27:08.126 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-09-16T19:27:08.128 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.151 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.041) 0:06:41.569 **** 2021-09-16T19:27:08.180 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-09-16T19:27:08.182 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.204 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.053) 0:06:41.622 **** 2021-09-16T19:27:08.221 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-09-16T19:27:08.223 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.249 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.044) 0:06:41.667 **** 2021-09-16T19:27:08.278 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-09-16T19:27:08.280 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.301 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.051) 0:06:41.719 **** 2021-09-16T19:27:08.318 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-09-16T19:27:08.320 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.343 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.041) 0:06:41.761 **** 2021-09-16T19:27:08.371 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-09-16T19:27:08.372 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.395 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.052) 0:06:41.813 **** 2021-09-16T19:27:08.412 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-09-16T19:27:08.414 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.437 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.041) 0:06:41.855 **** 2021-09-16T19:27:08.466 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-09-16T19:27:08.467 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.491 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.053) 0:06:41.908 **** 2021-09-16T19:27:08.508 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-09-16T19:27:08.510 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.533 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.042) 0:06:41.951 **** 2021-09-16T19:27:08.562 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-09-16T19:27:08.564 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.586 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.052) 0:06:42.004 **** 2021-09-16T19:27:08.614 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.637 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.050) 0:06:42.055 **** 2021-09-16T19:27:08.655 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-09-16T19:27:08.657 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.678 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.040) 0:06:42.096 **** 2021-09-16T19:27:08.706 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-09-16T19:27:08.708 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.730 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.051) 0:06:42.148 **** 2021-09-16T19:27:08.759 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-09-16T19:27:08.761 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.786 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.055) 0:06:42.203 **** 2021-09-16T19:27:08.803 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-09-16T19:27:08.805 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.828 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.042) 0:06:42.246 **** 2021-09-16T19:27:08.857 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-09-16T19:27:08.859 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.881 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.052) 0:06:42.299 **** 2021-09-16T19:27:08.919 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-16T19:27:08.921 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-09-16T19:27:08.943 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.062) 0:06:42.361 **** 2021-09-16T19:27:08.950 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-09-16T19:27:08.952 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:08.974 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:08 +0000 (0:00:00.030) 0:06:42.392 **** 2021-09-16T19:27:09.022 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-09-16T19:27:09.024 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.033 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-09-16T19:27:09.044 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.046 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} TASK [testnode : set_fact] ***************************************************** ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi109.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.070 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:09 +0000 (0:00:00.095) 0:06:42.488 **** 2021-09-16T19:27:09.076 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.078 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.105 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:09 +0000 (0:00:00.034) 0:06:42.522 **** 2021-09-16T19:27:09.142 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.153 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-09-16T19:27:09.164 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.166 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.175 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.177 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.186 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.214 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:09 +0000 (0:00:00.109) 0:06:42.631 **** 2021-09-16T19:27:09.335 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-09-16T19:27:09.337 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003682", "end": "2021-09-16 19:27:09.316292", "rc": 0, "start": "2021-09-16 19:27:09.312610", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_63133ca92c299bd23f40f94d370cf5381b0b1313/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_63133ca92c299bd23f40f94d370cf5381b0b1313/virtualenv/bin/teuthology-lock"]} 2021-09-16T19:27:09.361 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:09 +0000 (0:00:00.147) 0:06:42.779 **** 2021-09-16T19:27:09.429 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-09-16T19:27:09.431 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.454 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:09 +0000 (0:00:00.092) 0:06:42.872 **** 2021-09-16T19:27:09.461 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-09-16T19:27:09.462 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003059", "end": "2021-09-16 19:27:09.416536", "rc": 0, "start": "2021-09-16 19:27:09.413477", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_63133ca92c299bd23f40f94d370cf5381b0b1313/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_63133ca92c299bd23f40f94d370cf5381b0b1313/virtualenv/bin/teuthology-lock"]} 2021-09-16T19:27:09.490 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:09 +0000 (0:00:00.035) 0:06:42.908 **** 2021-09-16T19:27:09.517 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-09-16T19:27:09.519 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.541 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:09 +0000 (0:00:00.051) 0:06:42.959 **** 2021-09-16T19:27:09.560 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-09-16T19:27:09.561 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.586 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:09 +0000 (0:00:00.045) 0:06:43.004 **** 2021-09-16T19:27:09.625 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-09-16T19:27:09.627 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.649 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:09 +0000 (0:00:00.062) 0:06:43.067 **** 2021-09-16T19:27:09.668 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-09-16T19:27:09.669 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.691 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:09 +0000 (0:00:00.041) 0:06:43.109 **** 2021-09-16T19:27:09.718 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-09-16T19:27:09.719 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.741 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:09 +0000 (0:00:00.050) 0:06:43.159 **** 2021-09-16T19:27:09.759 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-09-16T19:27:09.761 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.783 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:09 +0000 (0:00:00.041) 0:06:43.201 **** 2021-09-16T19:27:09.852 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-09-16T19:27:09.854 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:09.874 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:09 +0000 (0:00:00.091) 0:06:43.292 **** 2021-09-16T19:27:12.804 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-09-16T19:27:12.806 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-16T19:27:12.829 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:12 +0000 (0:00:02.952) 0:06:46.245 **** 2021-09-16T19:27:12.845 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-16T19:27:12.869 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:12 +0000 (0:00:00.041) 0:06:46.287 **** 2021-09-16T19:27:12.907 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-09-16T19:27:12.909 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320678912, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129072, 'block_used': 12966036, 'inode_total': 61054976, 'inode_available': 60985436, 'inode_used': 69540, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129072, "block_size": 4096, "block_total": 240095108, "block_used": 12966036, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985436, "inode_total": 61054976, "inode_used": 69540, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320678912, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}} 2021-09-16T19:27:12.932 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:12 +0000 (0:00:00.063) 0:06:46.350 **** 2021-09-16T19:27:12.951 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320678912, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129072, 'block_used': 12966036, 'inode_total': 61054976, 'inode_available': 60985436, 'inode_used': 69540, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129072, "block_size": 4096, "block_total": 240095108, "block_used": 12966036, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985436, "inode_total": 61054976, "inode_used": 69540, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320678912, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}} 2021-09-16T19:27:12.976 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:12 +0000 (0:00:00.043) 0:06:46.394 **** 2021-09-16T19:27:13.171 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-09-16T19:27:13.173 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005100", "end": "2021-09-16 19:27:13.146025", "rc": 0, "start": "2021-09-16 19:27:13.140925", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:13.194 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:13 +0000 (0:00:00.218) 0:06:46.612 **** 2021-09-16T19:27:13.213 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005246", "end": "2021-09-16 19:27:13.187595", "rc": 0, "start": "2021-09-16 19:27:13.182349", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:13.238 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:13 +0000 (0:00:00.043) 0:06:46.655 **** 2021-09-16T19:27:13.275 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-09-16T19:27:13.277 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320678912, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129072, 'block_used': 12966036, 'inode_total': 61054976, 'inode_available': 60985436, 'inode_used': 69540, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129072, "block_size": 4096, "block_total": 240095108, "block_used": 12966036, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985436, "inode_total": 61054976, "inode_used": 69540, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320678912, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:13.300 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:13 +0000 (0:00:00.062) 0:06:46.718 **** 2021-09-16T19:27:13.317 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320678912, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129072, 'block_used': 12966036, 'inode_total': 61054976, 'inode_available': 60985436, 'inode_used': 69540, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129072, "block_size": 4096, "block_total": 240095108, "block_used": 12966036, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985436, "inode_total": 61054976, "inode_used": 69540, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320678912, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:13.342 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:13 +0000 (0:00:00.042) 0:06:46.760 **** 2021-09-16T19:27:13.369 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-09-16T19:27:13.393 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:13 +0000 (0:00:00.050) 0:06:46.811 **** 2021-09-16T19:27:13.434 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:13 +0000 (0:00:00.040) 0:06:46.851 **** 2021-09-16T19:27:14.799 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-09-16T19:27:14.801 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdcxr50wFNQjWhJ39Icd7LIGgui1h25yQbI'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.158744", "end": "2021-09-16 19:27:14.771331", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdcxr50wFNQjWhJ39Icd7LIGgui1h25yQbI"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:13.612587", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-16T19:27:14.874 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAx95aR9rnfHcjthRxeWJWg0P0n5ULrHMG'], 'uuids': ['0d160067-7a5a-4a69-a393-1bc80d1223c1'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.186812", "end": "2021-09-16 19:27:14.844945", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAx95aR9rnfHcjthRxeWJWg0P0n5ULrHMG"], "labels": [], "masters": [], "uuids": ["0d160067-7a5a-4a69-a393-1bc80d1223c1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:13.658133", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-16T19:27:16.060 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-kuVZYZ-kYpj-8PfN-IcO0-Jy7e-K0A2-SGRZGe', 'nvme-INTEL_SSDPEDMD400G4_PHFT6204016X400BGN', 'nvme-nvme.8086-50484654363230343031365834303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.069732", "end": "2021-09-16 19:27:16.027928", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-kuVZYZ-kYpj-8PfN-IcO0-Jy7e-K0A2-SGRZGe", "nvme-INTEL_SSDPEDMD400G4_PHFT6204016X400BGN", "nvme-nvme.8086-50484654363230343031365834303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:14.958196", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-09-16T19:27:16.062 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-16T19:27:16.124 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-8FrXnZ-oGcD-1vyp-2A3S-G218-8VBs-gt8cAF', 'nvme-INTEL_SSDPED1D480GA_PHMB747500JS480DGN', 'nvme-nvme.8086-50484d423734373530304a5334383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.057201", "end": "2021-09-16 19:27:16.095098", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-8FrXnZ-oGcD-1vyp-2A3S-G218-8VBs-gt8cAF", "nvme-INTEL_SSDPED1D480GA_PHMB747500JS480DGN", "nvme-nvme.8086-50484d423734373530304a5334383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:15.037897", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2021-09-16T19:27:16.125 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-16T19:27:17.290 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdcoUE0wHmlvroBcR6R2lvaOw6fgq4MRUDk'], 'uuids': ['221f8f75-8030-491b-9858-e9455bbd4bf7'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.044284", "end": "2021-09-16 19:27:17.255561", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdcoUE0wHmlvroBcR6R2lvaOw6fgq4MRUDk"], "labels": [], "masters": [], "uuids": ["221f8f75-8030-491b-9858-e9455bbd4bf7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:16.211277", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-16T19:27:17.384 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAa3zG1EU9JILzwr7yizegpqyujdGTjwGg'], 'uuids': ['50d923a2-8dc4-47bc-9d85-09e65eb914b2'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.068832", "end": "2021-09-16 19:27:17.353658", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAa3zG1EU9JILzwr7yizegpqyujdGTjwGg"], "labels": [], "masters": [], "uuids": ["50d923a2-8dc4-47bc-9d85-09e65eb914b2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:16.284826", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-16T19:27:18.523 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdcrjkC2XoA38fuejynBh2APf0GNoTxBL8L'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.051966", "end": "2021-09-16 19:27:18.491154", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdcrjkC2XoA38fuejynBh2APf0GNoTxBL8L"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:17.439188", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-16T19:27:18.650 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAinfTmScKebMwb5Ur2JpDea8ZkLB6adP9'], 'uuids': ['e1ae8cbb-f685-4ec6-b0c7-26a9ae41dabf'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.078811", "end": "2021-09-16 19:27:18.618885", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAinfTmScKebMwb5Ur2JpDea8ZkLB6adP9"], "labels": [], "masters": [], "uuids": ["e1ae8cbb-f685-4ec6-b0c7-26a9ae41dabf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:17.540074", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-16T19:27:19.757 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdcV4jgohtLizApJ1PXxJnO06rrs6NcFhrc'], 'uuids': ['ed2f288e-6a5f-4669-a4c4-40810fa3bd84'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.054673", "end": "2021-09-16 19:27:19.730782", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdcV4jgohtLizApJ1PXxJnO06rrs6NcFhrc"], "labels": [], "masters": [], "uuids": ["ed2f288e-6a5f-4669-a4c4-40810fa3bd84"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:18.676109", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-16T19:27:19.770 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QPMH', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPMH', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPMH', 'scsi-35000c50091e0bc82', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QPMH', 'wwn-0x5000c50091e0bc82'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QPMH', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e0bc82', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QPMH-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPMH-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPMH-part1', 'scsi-35000c50091e0bc82-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QPMH-part1', 'wwn-0x5000c50091e0bc82-part1'], 'uuids': ['bc802139-74bd-43cf-840d-e30da2bb57f8'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QPMH", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPMH", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPMH", "scsi-35000c50091e0bc82", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QPMH", "wwn-0x5000c50091e0bc82"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QPMH-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPMH-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-16T19:27:19.771 INFO:teuthology.task.ansible.out:Z1W5QPMH-part1", "scsi-35000c50091e0bc82-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QPMH-part1", "wwn-0x5000c50091e0bc82-part1"], "labels": [], "masters": [], "uuids": ["bc802139-74bd-43cf-840d-e30da2bb57f8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QPMH", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e0bc82"}}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:19.885 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAGWQwSW5F97CCiecOpMoGkLNlC7Pl0nom'], 'uuids': ['f4376fe3-9e8e-4b1e-ab9e-1681fd81de94'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.045377", "end": "2021-09-16 19:27:19.851941", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAGWQwSW5F97CCiecOpMoGkLNlC7Pl0nom"], "labels": [], "masters": [], "uuids": ["f4376fe3-9e8e-4b1e-ab9e-1681fd81de94"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:18.806564", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-16T19:27:19.886 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QARN', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QARN', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QARN', 'scsi-35000c50091e0b046', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QARN', 'wwn-0x5000c50091e0b046'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QARN', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e0b046', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QARN-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QARN-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QARN-part1', 'scsi-35000c50091e0b046-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QARN-part1', 'wwn-0x5000c50091e0b046-part1'], 'uuids': ['bc802139-74bd-43cf-840d-e30da2bb57f8'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QARN", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QARN", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QARN", "scsi-35000c50091e0b046", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QARN", "wwn-0x5000c50091e0b046"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QARN-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QARN-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-16T19:27:19.888 INFO:teuthology.task.ansible.out:Z1W5QARN-part1", "scsi-35000c50091e0b046-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QARN-part1", "wwn-0x5000c50091e0b046-part1"], "labels": [], "masters": [], "uuids": ["bc802139-74bd-43cf-840d-e30da2bb57f8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QARN", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e0b046"}}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:21.037 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdc20v5UHM68MDIGBVmtURdnm0BfhojEy43'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.081172", "end": "2021-09-16 19:27:21.005638", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdc20v5UHM68MDIGBVmtURdnm0BfhojEy43"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:19.924466", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-16T19:27:21.065 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:21 +0000 (0:00:07.631) 0:06:54.483 **** 2021-09-16T19:27:21.125 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAxd0xjf2zPXysoW3vlpvYhh83VooGYoQk'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.052303", "end": "2021-09-16 19:27:21.095594", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAxd0xjf2zPXysoW3vlpvYhh83VooGYoQk"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:20.043291", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-16T19:27:21.154 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:21 +0000 (0:00:00.088) 0:06:54.572 **** 2021-09-16T19:27:21.473 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-16T19:27:21.474 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdcxr50wFNQjWhJ39Icd7LIGgui1h25yQbI'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.144377", "end": "2021-09-16 19:27:21.448304", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdcxr50wFNQjWhJ39Icd7LIGgui1h25yQbI"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:21.303927", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.141669 s, 814 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.141669 s, 814 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:21.518 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAx95aR9rnfHcjthRxeWJWg0P0n5ULrHMG'], 'uuids': ['0d160067-7a5a-4a69-a393-1bc80d1223c1'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.101815", "end": "2021-09-16 19:27:21.487374", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAx95aR9rnfHcjthRxeWJWg0P0n5ULrHMG"], "labels": [], "masters": [], "uuids": ["0d160067-7a5a-4a69-a393-1bc80d1223c1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:21.385559", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0989921 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0989921 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:21.686 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-kuVZYZ-kYpj-8PfN-IcO0-Jy7e-K0A2-SGRZGe', 'nvme-INTEL_SSDPEDMD400G4_PHFT6204016X400BGN', 'nvme-nvme.8086-50484654363230343031365834303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039828", "end": "2021-09-16 19:27:21.660917", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-kuVZYZ-kYpj-8PfN-IcO0-Jy7e-K0A2-SGRZGe", "nvme-INTEL_SSDPEDMD400G4_PHFT6204016X400BGN", "nvme-nvme.8086-50484654363230343031365834303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:21.621089", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0370988 s, 3.1 GB/s", "std 2021-09-16T19:27:21.688 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0370988 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:21.730 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-8FrXnZ-oGcD-1vyp-2A3S-G218-8VBs-gt8cAF', 'nvme-INTEL_SSDPED1D480GA_PHMB747500JS480DGN', 'nvme-nvme.8086-50484d423734373530304a5334383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039854", "end": "2021-09-16 19:27:21.698846", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-8FrXnZ-oGcD-1vyp-2A3S-G218-8VBs-gt8cAF", "nvme-INTEL_SSDPED1D480GA_PHMB747500JS480DGN", "nvme-nvme.8086-50484d423734373530304a5334383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:21.658992", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0370016 s, 3.1 GB/s", "stderr_lines": [" 2021-09-16T19:27:21.732 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0370016 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:22.003 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAa3zG1EU9JILzwr7yizegpqyujdGTjwGg'], 'uuids': ['50d923a2-8dc4-47bc-9d85-09e65eb914b2'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.099561", "end": "2021-09-16 19:27:21.977573", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAa3zG1EU9JILzwr7yizegpqyujdGTjwGg"], "labels": [], "masters": [], "uuids": ["50d923a2-8dc4-47bc-9d85-09e65eb914b2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:21.878012", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0968674 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0968674 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:22.015 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdcoUE0wHmlvroBcR6R2lvaOw6fgq4MRUDk'], 'uuids': ['221f8f75-8030-491b-9858-e9455bbd4bf7'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.149550", "end": "2021-09-16 19:27:21.992538", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdcoUE0wHmlvroBcR6R2lvaOw6fgq4MRUDk"], "labels": [], "masters": [], "uuids": ["221f8f75-8030-491b-9858-e9455bbd4bf7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:21.842988", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.146799 s, 786 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.146799 s, 786 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:22.288 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAinfTmScKebMwb5Ur2JpDea8ZkLB6adP9'], 'uuids': ['e1ae8cbb-f685-4ec6-b0c7-26a9ae41dabf'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.101554", "end": "2021-09-16 19:27:22.255668", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAinfTmScKebMwb5Ur2JpDea8ZkLB6adP9"], "labels": [], "masters": [], "uuids": ["e1ae8cbb-f685-4ec6-b0c7-26a9ae41dabf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:22.154114", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0988441 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0988441 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:22.341 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdcrjkC2XoA38fuejynBh2APf0GNoTxBL8L'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.150882", "end": "2021-09-16 19:27:22.315614", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdcrjkC2XoA38fuejynBh2APf0GNoTxBL8L"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:22.164732", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148195 s, 778 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148195 s, 778 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:22.550 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAGWQwSW5F97CCiecOpMoGkLNlC7Pl0nom'], 'uuids': ['f4376fe3-9e8e-4b1e-ab9e-1681fd81de94'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.099582", "end": "2021-09-16 19:27:22.525936", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAGWQwSW5F97CCiecOpMoGkLNlC7Pl0nom"], "labels": [], "masters": [], "uuids": ["f4376fe3-9e8e-4b1e-ab9e-1681fd81de94"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:22.426354", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0969345 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0969345 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:22.562 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QARN', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QARN', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QARN', 'scsi-35000c50091e0b046', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QARN', 'wwn-0x5000c50091e0b046'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QARN', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e0b046', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QARN-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QARN-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QARN-part1', 'scsi-35000c50091e0b046-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QARN-part1', 'wwn-0x5000c50091e0b046-part1'], 'uuids': ['bc802139-74bd-43cf-840d-e30da2bb57f8'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QARN", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QARN", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QARN", "scsi-35000c50091e0b046", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QARN", "wwn-0x5000c50091e0b046"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QARN-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QARN-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-16T19:27:22.564 INFO:teuthology.task.ansible.out:Z1W5QARN-part1", "scsi-35000c50091e0b046-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QARN-part1", "wwn-0x5000c50091e0b046-part1"], "labels": [], "masters": [], "uuids": ["bc802139-74bd-43cf-840d-e30da2bb57f8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QARN", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e0b046"}}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:22.668 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdcV4jgohtLizApJ1PXxJnO06rrs6NcFhrc'], 'uuids': ['ed2f288e-6a5f-4669-a4c4-40810fa3bd84'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.146759", "end": "2021-09-16 19:27:22.639257", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdcV4jgohtLizApJ1PXxJnO06rrs6NcFhrc"], "labels": [], "masters": [], "uuids": ["ed2f288e-6a5f-4669-a4c4-40810fa3bd84"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:22.492498", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144067 s, 801 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144067 s, 801 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:22.669 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QPMH', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPMH', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPMH', 'scsi-35000c50091e0bc82', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QPMH', 'wwn-0x5000c50091e0bc82'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QPMH', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e0bc82', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QPMH-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPMH-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPMH-part1', 'scsi-35000c50091e0bc82-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QPMH-part1', 'wwn-0x5000c50091e0bc82-part1'], 'uuids': ['bc802139-74bd-43cf-840d-e30da2bb57f8'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QPMH", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPMH", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QPMH", "scsi-35000c50091e0bc82", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QPMH", "wwn-0x5000c50091e0bc82"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QPMH-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QPMH-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-16T19:27:22.671 INFO:teuthology.task.ansible.out:Z1W5QPMH-part1", "scsi-35000c50091e0bc82-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QPMH-part1", "wwn-0x5000c50091e0bc82-part1"], "labels": [], "masters": [], "uuids": ["bc802139-74bd-43cf-840d-e30da2bb57f8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QPMH", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e0bc82"}}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:22.827 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAxd0xjf2zPXysoW3vlpvYhh83VooGYoQk'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.100475", "end": "2021-09-16 19:27:22.805568", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-dDtVdpRfLSh4OeEgY9jp7PtjGRurx8zAxd0xjf2zPXysoW3vlpvYhh83VooGYoQk"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:22.705093", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0977458 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0977458 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:22.867 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:22 +0000 (0:00:01.712) 0:06:56.285 **** 2021-09-16T19:27:22.980 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdc20v5UHM68MDIGBVmtURdnm0BfhojEy43'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.145446", "end": "2021-09-16 19:27:22.959299", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-PW8jc2rd9EytnsjqOhH06HW52W30lLdc20v5UHM68MDIGBVmtURdnm0BfhojEy43"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-16 19:27:22.813853", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142763 s, 808 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142763 s, 808 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:23.021 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:23 +0000 (0:00:00.154) 0:06:56.439 **** 2021-09-16T19:27:23.175 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-09-16T19:27:23.177 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.068360", "end": "2021-09-16 19:27:23.094880", "rc": 0, "start": "2021-09-16 19:27:23.026520", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:23.201 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:23 +0000 (0:00:00.179) 0:06:56.619 **** 2021-09-16T19:27:23.293 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.045580", "end": "2021-09-16 19:27:23.227674", "rc": 0, "start": "2021-09-16 19:27:23.182094", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:23.318 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:23 +0000 (0:00:00.117) 0:06:56.736 **** 2021-09-16T19:27:23.432 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-09-16T19:27:23.434 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.051868", "end": "2021-09-16 19:27:23.413758", "rc": 0, "start": "2021-09-16 19:27:23.361890", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:23.458 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:23 +0000 (0:00:00.139) 0:06:56.875 **** 2021-09-16T19:27:23.528 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-09-16T19:27:23.590 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:23 +0000 (0:00:00.132) 0:06:57.008 **** 2021-09-16T19:27:23.598 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-09-16T19:27:23.601 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.045869", "end": "2021-09-16 19:27:23.524617", "rc": 0, "start": "2021-09-16 19:27:23.478748", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:23.624 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:23 +0000 (0:00:00.033) 0:06:57.042 **** 2021-09-16T19:27:23.632 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-09-16T19:27:23.634 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:23.660 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:23 +0000 (0:00:00.035) 0:06:57.078 **** 2021-09-16T19:27:23.666 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-09-16T19:27:23.693 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:23 +0000 (0:00:00.032) 0:06:57.111 **** 2021-09-16T19:27:23.700 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-09-16T19:27:23.702 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:23.724 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:23 +0000 (0:00:00.031) 0:06:57.142 **** 2021-09-16T19:27:23.731 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-09-16T19:27:23.733 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:23.757 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:23 +0000 (0:00:00.032) 0:06:57.175 **** 2021-09-16T19:27:23.764 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-09-16T19:27:23.766 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:23.791 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:23 +0000 (0:00:00.033) 0:06:57.208 **** 2021-09-16T19:27:23.798 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-09-16T19:27:23.800 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:23.822 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:23 +0000 (0:00:00.031) 0:06:57.240 **** 2021-09-16T19:27:23.829 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-09-16T19:27:23.830 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:23.852 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:23 +0000 (0:00:00.030) 0:06:57.270 **** 2021-09-16T19:27:23.859 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-09-16T19:27:23.860 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:23.886 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:23 +0000 (0:00:00.033) 0:06:57.304 **** 2021-09-16T19:27:23.914 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-09-16T19:27:23.916 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:23.939 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:23 +0000 (0:00:00.052) 0:06:57.357 **** 2021-09-16T19:27:24.042 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-09-16T19:27:24.044 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-16T19:27:24.066 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:24 +0000 (0:00:00.127) 0:06:57.484 **** 2021-09-16T19:27:24.115 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-09-16T19:27:24.117 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320678912, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129072, 'block_used': 12966036, 'inode_total': 61054976, 'inode_available': 60985436, 'inode_used': 69540, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129072, "block_size": 4096, "block_total": 240095108, "block_used": 12966036, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985436, "inode_total": 61054976, "inode_used": 69540, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320678912, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:24.141 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:24 +0000 (0:00:00.075) 0:06:57.559 **** 2021-09-16T19:27:24.148 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-09-16T19:27:24.150 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-16T19:27:24.173 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:24 +0000 (0:00:00.031) 0:06:57.591 **** 2021-09-16T19:27:24.179 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-09-16T19:27:24.181 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:24.204 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:24 +0000 (0:00:00.031) 0:06:57.622 **** 2021-09-16T19:27:24.222 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi109.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320678912, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129072, 'block_used': 12966036, 'inode_total': 61054976, 'inode_available': 60985436, 'inode_used': 69540, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129072, "block_size": 4096, "block_total": 240095108, "block_used": 12966036, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985436, "inode_total": 61054976, "inode_used": 69540, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320678912, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:24.245 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:24 +0000 (0:00:00.040) 0:06:57.663 **** 2021-09-16T19:27:24.251 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-16T19:27:24.253 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:24.275 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:24 +0000 (0:00:00.029) 0:06:57.693 **** 2021-09-16T19:27:24.281 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-09-16T19:27:24.283 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:24.305 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:24 +0000 (0:00:00.030) 0:06:57.723 **** 2021-09-16T19:27:24.313 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-16T19:27:24.315 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:24.337 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:24 +0000 (0:00:00.031) 0:06:57.755 **** 2021-09-16T19:27:24.343 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-16T19:27:24.345 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:24.372 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:24 +0000 (0:00:00.035) 0:06:57.790 **** 2021-09-16T19:27:24.380 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-09-16T19:27:24.381 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:24.404 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:24 +0000 (0:00:00.031) 0:06:57.821 **** 2021-09-16T19:27:24.410 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-16T19:27:24.412 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:24.434 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:24 +0000 (0:00:00.030) 0:06:57.852 **** 2021-09-16T19:27:24.441 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-09-16T19:27:24.443 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:24.465 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:24 +0000 (0:00:00.030) 0:06:57.882 **** 2021-09-16T19:27:24.471 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-09-16T19:27:24.472 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:24.495 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:24 +0000 (0:00:00.030) 0:06:57.913 **** 2021-09-16T19:27:24.504 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-09-16T19:27:24.505 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:24.531 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:24 +0000 (0:00:00.035) 0:06:57.948 **** 2021-09-16T19:27:24.538 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-09-16T19:27:24.540 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:24.567 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:24 +0000 (0:00:00.035) 0:06:57.984 **** 2021-09-16T19:27:24.597 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-09-16T19:27:24.598 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:24.619 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:24 +0000 (0:00:00.051) 0:06:58.036 **** 2021-09-16T19:27:25.203 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-09-16T19:27:25.204 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-09-16T19:27:25.226 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:25 +0000 (0:00:00.607) 0:06:58.644 **** 2021-09-16T19:27:25.235 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-09-16T19:27:25.263 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:25 +0000 (0:00:00.036) 0:06:58.681 **** 2021-09-16T19:27:25.856 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-09-16T19:27:25.857 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-16T19:27:25.858 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-16T19:27:26.202 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-16T19:27:26.223 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-16T19:27:26.555 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-16T19:27:26.557 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-16T19:27:26.889 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-16T19:27:26.901 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-16T19:27:27.214 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-16T19:27:27.238 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:27 +0000 (0:00:01.975) 0:07:00.656 **** 2021-09-16T19:27:27.246 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-16T19:27:27.272 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:27 +0000 (0:00:00.034) 0:07:00.690 **** 2021-09-16T19:27:27.425 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-09-16T19:27:27.427 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-09-16T19:27:27.449 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:27 +0000 (0:00:00.176) 0:07:00.866 **** 2021-09-16T19:27:27.467 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-09-16T19:27:27.490 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:27 +0000 (0:00:00.040) 0:07:00.907 **** 2021-09-16T19:27:27.663 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-09-16T19:27:27.665 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-16T19:27:27.696 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-16T19:27:27.834 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-16T19:27:27.867 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-16T19:27:28.003 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-16T19:27:28.035 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-16T19:27:28.182 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-16T19:27:28.193 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:28.219 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:28 +0000 (0:00:00.729) 0:07:01.637 **** 2021-09-16T19:27:28.227 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-16T19:27:28.239 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:28.264 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:28 +0000 (0:00:00.044) 0:07:01.682 **** 2021-09-16T19:27:28.271 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-09-16T19:27:28.273 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:28.295 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:28 +0000 (0:00:00.030) 0:07:01.712 **** 2021-09-16T19:27:28.301 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:28.328 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:28 +0000 (0:00:00.032) 0:07:01.745 **** 2021-09-16T19:27:28.335 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-09-16T19:27:28.337 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:28.360 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:28 +0000 (0:00:00.032) 0:07:01.778 **** 2021-09-16T19:27:28.367 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:28.392 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:28 +0000 (0:00:00.032) 0:07:01.810 **** 2021-09-16T19:27:28.560 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-09-16T19:27:28.561 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-16T19:27:28.584 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:28 +0000 (0:00:00.191) 0:07:02.001 **** 2021-09-16T19:27:28.601 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-16T19:27:28.624 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:28 +0000 (0:00:00.039) 0:07:02.041 **** 2021-09-16T19:27:28.630 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-09-16T19:27:28.632 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:28.658 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:28 +0000 (0:00:00.034) 0:07:02.076 **** 2021-09-16T19:27:28.665 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:28.689 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:28 +0000 (0:00:00.030) 0:07:02.106 **** 2021-09-16T19:27:29.124 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-09-16T19:27:29.126 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true} 2021-09-16T19:27:29.163 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:29 +0000 (0:00:00.474) 0:07:02.581 **** 2021-09-16T19:27:29.171 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true} 2021-09-16T19:27:29.201 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:29 +0000 (0:00:00.038) 0:07:02.619 **** 2021-09-16T19:27:29.719 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-09-16T19:27:29.720 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-09-16T19:27:29.741 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:29 +0000 (0:00:00.539) 0:07:03.159 **** 2021-09-16T19:27:29.761 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:29 +0000 (0:00:00.019) 0:07:03.179 **** 2021-09-16T19:27:29.778 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-09-16T19:27:29.780 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:29.800 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:29 +0000 (0:00:00.039) 0:07:03.218 **** 2021-09-16T19:27:29.807 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:29.830 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:29 +0000 (0:00:00.029) 0:07:03.247 **** 2021-09-16T19:27:29.835 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-09-16T19:27:29.836 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:29.859 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:29 +0000 (0:00:00.028) 0:07:03.276 **** 2021-09-16T19:27:29.865 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:29.887 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:29 +0000 (0:00:00.028) 0:07:03.305 **** 2021-09-16T19:27:29.892 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-09-16T19:27:29.894 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:29.919 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:29 +0000 (0:00:00.032) 0:07:03.337 **** 2021-09-16T19:27:29.926 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:29.950 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:29 +0000 (0:00:00.030) 0:07:03.368 **** 2021-09-16T19:27:29.956 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-09-16T19:27:29.958 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:29.981 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:29 +0000 (0:00:00.030) 0:07:03.399 **** 2021-09-16T19:27:29.988 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.011 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.030) 0:07:03.429 **** 2021-09-16T19:27:30.020 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-09-16T19:27:30.021 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.044 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.032) 0:07:03.462 **** 2021-09-16T19:27:30.051 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.074 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.030) 0:07:03.492 **** 2021-09-16T19:27:30.081 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-09-16T19:27:30.083 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.105 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.031) 0:07:03.523 **** 2021-09-16T19:27:30.113 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.136 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.030) 0:07:03.554 **** 2021-09-16T19:27:30.142 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-09-16T19:27:30.143 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.165 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.029) 0:07:03.583 **** 2021-09-16T19:27:30.173 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.196 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.030) 0:07:03.614 **** 2021-09-16T19:27:30.203 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-09-16T19:27:30.205 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.228 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.031) 0:07:03.646 **** 2021-09-16T19:27:30.236 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.260 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.031) 0:07:03.677 **** 2021-09-16T19:27:30.266 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-09-16T19:27:30.268 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.290 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.030) 0:07:03.708 **** 2021-09-16T19:27:30.296 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.319 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.029) 0:07:03.737 **** 2021-09-16T19:27:30.325 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-09-16T19:27:30.327 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.350 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.030) 0:07:03.768 **** 2021-09-16T19:27:30.357 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.380 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.030) 0:07:03.798 **** 2021-09-16T19:27:30.386 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-09-16T19:27:30.388 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.410 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.030) 0:07:03.828 **** 2021-09-16T19:27:30.422 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.445 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.034) 0:07:03.863 **** 2021-09-16T19:27:30.451 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-09-16T19:27:30.453 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.475 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.030) 0:07:03.893 **** 2021-09-16T19:27:30.482 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.509 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.033) 0:07:03.926 **** 2021-09-16T19:27:30.516 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-09-16T19:27:30.518 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.543 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.034) 0:07:03.961 **** 2021-09-16T19:27:30.552 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.577 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.034) 0:07:03.995 **** 2021-09-16T19:27:30.585 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-09-16T19:27:30.586 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.615 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.037) 0:07:04.032 **** 2021-09-16T19:27:30.624 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.647 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.032) 0:07:04.065 **** 2021-09-16T19:27:30.654 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-09-16T19:27:30.656 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.683 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.036) 0:07:04.101 **** 2021-09-16T19:27:30.691 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:30.716 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:30 +0000 (0:00:00.033) 0:07:04.134 **** 2021-09-16T19:27:31.149 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-09-16T19:27:31.151 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-09-16T19:27:31.172 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:31 +0000 (0:00:00.455) 0:07:04.590 **** 2021-09-16T19:27:31.190 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-09-16T19:27:31.214 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:31 +0000 (0:00:00.042) 0:07:04.632 **** 2021-09-16T19:27:31.463 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-09-16T19:27:31.464 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-09-16 19:18:49 UTC", "ActiveEnterTimestampMonotonic": "15543444", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket system.slice ntpdate.service sysinit.target basic.target sntp.service systemd-tmpfiles-setup.service ntpd.service tmp.mount -.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-09-16 19:18:47 UTC", "AssertTimestampMonotonic": "13875353", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-09-16 19:18:47 UTC", "ConditionTimestampMonotonic": "13875317", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.tar 2021-09-16T19:27:31.466 INFO:teuthology.task.ansible.out:get ntpd.service systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "886", "ExecMainStartTimestamp": "Thu 2021-09-16 19:18:49 UTC", "ExecMainStartTimestampMonotonic": "15430097", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-09-16 19:18:47 UTC", "InactiveExitTimestampMonotonic": "13919376", "InvocationID": "ce2f7335b55c4b05837baa7a419a0f38", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinit 2021-09-16T19:27:31.467 INFO:teuthology.task.ansible.out:y", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127240", "LimitNPROCSoft": "127240", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127240", "LimitSIGPENDINGSoft": "127240", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "886", "MemoryAccounting": "yes", "MemoryCurrent": "1253376", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target - 2021-09-16T19:27:31.469 INFO:teuthology.task.ansible.out:.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-09-16 19:18:49 UTC", "StateChangeTimestampMonotonic": "15543444", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203584", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-09-16 19:18:49 UTC", "WatchdogTimestampMonotonic": "15430316", "WatchdogUSec": "0"}} 2021-09-16T19:27:31.488 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:31 +0000 (0:00:00.274) 0:07:04.906 **** 2021-09-16T19:27:31.517 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-16T19:27:31.518 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} TASK [testnode : Make sure ntpd is running.] *********************************** 2021-09-16T19:27:31.520 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2021-09-16 19:19:28 UTC", "ActiveEnterTimestampMonotonic": "15729208", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "tmp.mount sysinit.target -.mount ntpdate.service systemd-journald.socket system.slice basic.target systemd-tmpfiles-setup.service sntp.service ntpd.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2021-09-16 19:19:26 UTC", "AssertTimestampMonotonic": "14241165", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2021-09-16 19:19:26 UTC", "ConditionTimestampMonotonic": "14241118", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.tar 2021-09-16T19:27:31.522 INFO:teuthology.task.ansible.out:get systemd-timesyncd.service ntpd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "893", "ExecMainStartTimestamp": "Thu 2021-09-16 19:19:28 UTC", "ExecMainStartTimestampMonotonic": "15545636", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2021-09-16 19:19:26 UTC", "InactiveExitTimestampMonotonic": "14302538", "InvocationID": "37d73717a1f641aabbc0f023a93d8e9e", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinit 2021-09-16T19:27:31.523 INFO:teuthology.task.ansible.out:y", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127239", "LimitNPROCSoft": "127239", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127239", "LimitSIGPENDINGSoft": "127239", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "893", "MemoryAccounting": "yes", "MemoryCurrent": "1249280", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system 2021-09-16T19:27:31.525 INFO:teuthology.task.ansible.out:.slice", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2021-09-16 19:19:28 UTC", "StateChangeTimestampMonotonic": "15729208", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203583", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2021-09-16 19:19:28 UTC", "WatchdogTimestampMonotonic": "15545654", "WatchdogUSec": "0"}} 2021-09-16T19:27:31.542 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:31 +0000 (0:00:00.053) 0:07:04.960 **** 2021-09-16T19:27:31.564 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:31 +0000 (0:00:00.021) 0:07:04.981 **** 2021-09-16T19:27:31.581 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-16T19:27:31.583 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-09-16T19:27:31.603 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:31 +0000 (0:00:00.039) 0:07:05.021 **** 2021-09-16T19:27:31.611 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-16T19:27:31.612 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-09-16T19:27:31.634 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:31 +0000 (0:00:00.031) 0:07:05.052 **** 2021-09-16T19:27:31.641 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:31.664 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:31 +0000 (0:00:00.030) 0:07:05.082 **** 2021-09-16T19:27:31.670 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-16T19:27:31.672 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-09-16T19:27:31.700 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:31 +0000 (0:00:00.035) 0:07:05.117 **** 2021-09-16T19:27:31.707 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-09-16T19:27:31.709 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:31.732 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:31 +0000 (0:00:00.032) 0:07:05.150 **** 2021-09-16T19:27:31.738 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-09-16T19:27:31.740 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:31.762 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:31 +0000 (0:00:00.030) 0:07:05.180 **** 2021-09-16T19:27:31.769 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-09-16T19:27:31.770 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:31.792 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:31 +0000 (0:00:00.029) 0:07:05.210 **** 2021-09-16T19:27:31.799 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-09-16T19:27:31.801 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:31.824 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:31 +0000 (0:00:00.031) 0:07:05.242 **** 2021-09-16T19:27:31.852 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-09-16T19:27:31.854 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:31.874 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:31 +0000 (0:00:00.050) 0:07:05.292 **** 2021-09-16T19:27:31.986 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-09-16T19:27:32.008 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:32 +0000 (0:00:00.133) 0:07:05.426 **** 2021-09-16T19:27:32.079 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-09-16T19:27:32.104 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:32 +0000 (0:00:00.096) 0:07:05.522 **** 2021-09-16T19:27:32.479 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-09-16T19:27:32.481 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-09-16T19:27:32.503 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:32 +0000 (0:00:00.399) 0:07:05.921 **** 2021-09-16T19:27:32.575 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-09-16T19:27:32.600 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:32 +0000 (0:00:00.096) 0:07:06.018 **** 2021-09-16T19:27:33.820 INFO:teuthology.task.ansible.out: 2021-09-16T19:27:33.822 INFO:teuthology.task.ansible.out:TASK [testnode : Install packages via pip] ************************************* ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-09-16T19:27:33.847 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:33 +0000 (0:00:01.245) 0:07:07.263 **** 2021-09-16T19:27:33.865 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-09-16T19:27:33.887 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:33 +0000 (0:00:00.041) 0:07:07.305 **** 2021-09-16T19:27:33.895 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-09-16T19:27:33.898 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:33.919 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:33 +0000 (0:00:00.031) 0:07:07.337 **** 2021-09-16T19:27:33.926 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:33.952 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:33 +0000 (0:00:00.033) 0:07:07.370 **** 2021-09-16T19:27:33.962 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-09-16T19:27:33.964 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-09-16T19:27:33.989 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:33 +0000 (0:00:00.036) 0:07:07.407 **** 2021-09-16T19:27:33.998 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-09-16T19:27:34.024 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:34 +0000 (0:00:00.034) 0:07:07.441 **** 2021-09-16T19:27:34.168 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-09-16T19:27:34.169 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-09-16T19:27:34.191 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:34 +0000 (0:00:00.167) 0:07:07.609 **** 2021-09-16T19:27:34.209 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-09-16T19:27:34.234 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:34 +0000 (0:00:00.042) 0:07:07.652 **** 2021-09-16T19:27:34.241 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-09-16T19:27:34.243 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:34.266 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:34 +0000 (0:00:00.031) 0:07:07.684 **** 2021-09-16T19:27:34.275 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:34.299 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:34 +0000 (0:00:00.033) 0:07:07.717 **** 2021-09-16T19:27:34.307 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-09-16T19:27:34.308 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-09-16T19:27:34.332 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:34 +0000 (0:00:00.032) 0:07:07.750 **** 2021-09-16T19:27:34.339 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-09-16T19:27:34.364 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:34 +0000 (0:00:00.032) 0:07:07.782 **** 2021-09-16T19:27:34.371 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-09-16T19:27:34.373 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:34.396 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:34 +0000 (0:00:00.031) 0:07:07.814 **** 2021-09-16T19:27:34.404 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:34.430 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:34 +0000 (0:00:00.033) 0:07:07.848 **** 2021-09-16T19:27:34.594 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-09-16T19:27:34.596 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-09-16T19:27:34.623 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:34 +0000 (0:00:00.193) 0:07:08.041 **** 2021-09-16T19:27:34.634 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-09-16T19:27:34.666 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:34 +0000 (0:00:00.042) 0:07:08.084 **** 2021-09-16T19:27:34.824 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi109.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-09-16T19:27:34.856 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:34 +0000 (0:00:00.189) 0:07:08.273 **** 2021-09-16T19:27:34.873 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-09-16T19:27:34.894 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:34 +0000 (0:00:00.038) 0:07:08.312 **** 2021-09-16T19:27:34.900 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-09-16T19:27:34.901 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:34.925 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:34 +0000 (0:00:00.031) 0:07:08.343 **** 2021-09-16T19:27:34.932 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:34.957 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:34 +0000 (0:00:00.031) 0:07:08.375 **** 2021-09-16T19:27:34.966 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-09-16T19:27:34.968 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:34.978 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:34.980 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:34.989 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:34.992 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.015 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.057) 0:07:08.432 **** 2021-09-16T19:27:35.023 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.024 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi118.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.025 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.052 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.037) 0:07:08.470 **** 2021-09-16T19:27:35.059 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-09-16T19:27:35.061 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.072 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.074 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi109.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.105 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.052) 0:07:08.523 **** 2021-09-16T19:27:35.115 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.116 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.117 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.117 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.118 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.142 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.036) 0:07:08.560 **** 2021-09-16T19:27:35.149 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-16T19:27:35.151 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.177 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.035) 0:07:08.595 **** 2021-09-16T19:27:35.185 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.212 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.034) 0:07:08.630 **** 2021-09-16T19:27:35.219 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-09-16T19:27:35.221 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.247 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.034) 0:07:08.665 **** 2021-09-16T19:27:35.254 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.278 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.030) 0:07:08.696 **** 2021-09-16T19:27:35.285 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-09-16T19:27:35.286 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.314 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.035) 0:07:08.732 **** 2021-09-16T19:27:35.321 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.348 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.034) 0:07:08.766 **** 2021-09-16T19:27:35.356 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-09-16T19:27:35.357 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.380 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.032) 0:07:08.798 **** 2021-09-16T19:27:35.389 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.413 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.032) 0:07:08.831 **** 2021-09-16T19:27:35.420 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-09-16T19:27:35.422 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.447 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.034) 0:07:08.865 **** 2021-09-16T19:27:35.455 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.481 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.032) 0:07:08.898 **** 2021-09-16T19:27:35.487 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-09-16T19:27:35.489 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.514 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.033) 0:07:08.931 **** 2021-09-16T19:27:35.522 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.548 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.034) 0:07:08.966 **** 2021-09-16T19:27:35.555 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-09-16T19:27:35.557 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.581 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.032) 0:07:08.998 **** 2021-09-16T19:27:35.587 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.612 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.030) 0:07:09.030 **** 2021-09-16T19:27:35.618 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-09-16T19:27:35.620 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.651 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.039) 0:07:09.069 **** 2021-09-16T19:27:35.663 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:35.687 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:35 +0000 (0:00:00.035) 0:07:09.105 **** 2021-09-16T19:27:35.859 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-09-16T19:27:35.861 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005408", "end": "2021-09-16 19:27:35.830720", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-09-16 19:27:35.825312", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:35.902 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005171", "end": "2021-09-16 19:27:35.870196", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-09-16 19:27:35.865025", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:36.028 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002951", "end": "2021-09-16 19:27:36.004403", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-09-16 19:27:36.001452", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:36.070 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003066", "end": "2021-09-16 19:27:36.042713", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-09-16 19:27:36.039647", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:36.206 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.003029", "end": "2021-09-16 19:27:36.174821", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-09-16 19:27:36.171792", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:36.238 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002784", "end": "2021-09-16 19:27:36.210352", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-09-16 19:27:36.207568", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:36.373 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003257", "end": "2021-09-16 19:27:36.349232", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-09-16 19:27:36.345975", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:36.398 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:36 +0000 (0:00:00.710) 0:07:09.816 **** 2021-09-16T19:27:36.407 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003270", "end": "2021-09-16 19:27:36.378152", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-09-16 19:27:36.374882", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:27:36.434 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:36 +0000 (0:00:00.036) 0:07:09.852 **** 2021-09-16T19:27:36.440 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-09-16T19:27:36.469 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:36 +0000 (0:00:00.035) 0:07:09.887 **** 2021-09-16T19:27:36.501 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:36 +0000 (0:00:00.031) 0:07:09.919 **** 2021-09-16T19:27:36.509 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-09-16T19:27:36.510 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:36.533 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:36.611 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-09-16T19:27:36.628 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:36 +0000 (0:00:00.127) 0:07:10.046 **** 2021-09-16T19:27:36.651 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:36 +0000 (0:00:00.022) 0:07:10.068 **** 2021-09-16T19:27:37.277 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi109.front.sepia.ceph.com] 2021-09-16T19:27:37.310 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] 2021-09-16T19:27:37.336 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:37 +0000 (0:00:00.685) 0:07:10.754 **** 2021-09-16T19:27:37.362 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-16T19:27:37.364 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:37.382 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:37 +0000 (0:00:00.046) 0:07:10.800 **** 2021-09-16T19:27:37.403 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:37 +0000 (0:00:00.020) 0:07:10.821 **** 2021-09-16T19:27:37.421 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-16T19:27:37.422 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-09-16T19:27:37.444 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:37 +0000 (0:00:00.040) 0:07:10.861 **** 2021-09-16T19:27:37.452 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-16T19:27:37.454 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:37.477 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:37 +0000 (0:00:00.033) 0:07:10.895 **** 2021-09-16T19:27:37.505 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-09-16T19:27:37.507 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} TASK [container-host : set_fact] *********************************************** ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-09-16T19:27:37.530 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:37 +0000 (0:00:00.052) 0:07:10.948 **** 2021-09-16T19:27:37.552 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:37 +0000 (0:00:00.021) 0:07:10.969 **** 2021-09-16T19:27:37.601 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-09-16T19:27:37.603 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-09-16T19:27:37.625 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:37 +0000 (0:00:00.073) 0:07:11.043 **** 2021-09-16T19:27:57.420 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-09-16T19:27:57.422 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: podman-docker-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.noarch", "Installed: podman-catatonit-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.x86_64", "Installed: podman-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.x86_64", "Removed: podman-3.0.1-6.module+el8.4.0+10607+f4da7515.x86_64", "Removed: podman-catatonit-3.0.1-6.module+el8.4.0+10607+f4da7515.x86_64", "Removed: podman-docker-3.0.1-6.module+el8.4.0+10607+f4da7515.noarch"]} 2021-09-16T19:27:57.441 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:57 +0000 (0:00:19.816) 0:07:30.859 **** 2021-09-16T19:27:57.449 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: podman-docker-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.noarch", "Installed: podman-catatonit-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.x86_64", "Installed: podman-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.x86_64", "Removed: podman-3.0.1-6.module+el8.4.0+10607+f4da7515.x86_64", "Removed: podman-catatonit-3.0.1-6.module+el8.4.0+10607+f4da7515.x86_64", "Removed: podman-docker-3.0.1-6.module+el8.4.0+10607+f4da7515.noarch"]} 2021-09-16T19:27:57.474 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:57 +0000 (0:00:00.032) 0:07:30.891 **** 2021-09-16T19:27:57.481 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-16T19:27:57.483 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-09-16T19:27:57.505 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:57 +0000 (0:00:00.031) 0:07:30.923 **** 2021-09-16T19:27:57.512 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-09-16T19:27:57.535 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:57 +0000 (0:00:00.030) 0:07:30.953 **** 2021-09-16T19:27:57.542 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-16T19:27:57.544 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:57.565 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:57 +0000 (0:00:00.030) 0:07:30.983 **** 2021-09-16T19:27:57.571 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:27:57.596 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:57 +0000 (0:00:00.030) 0:07:31.013 **** 2021-09-16T19:27:57.780 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-09-16T19:27:57.781 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-16T19:27:57.813 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-16T19:27:57.949 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-16T19:27:57.975 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:57 +0000 (0:00:00.379) 0:07:31.393 **** 2021-09-16T19:27:57.992 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-16T19:27:58.016 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:58 +0000 (0:00:00.041) 0:07:31.434 **** 2021-09-16T19:27:58.452 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-09-16T19:27:58.454 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-16T19:27:58.495 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-16T19:27:58.871 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-16T19:27:58.902 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:58 +0000 (0:00:00.885) 0:07:32.320 **** 2021-09-16T19:27:58.919 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-16T19:27:58.944 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:27:58 +0000 (0:00:00.042) 0:07:32.362 **** 2021-09-16T19:28:12.632 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** 2021-09-16T19:28:12.634 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-16T19:28:12.655 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:28:12 +0000 (0:00:13.710) 0:07:46.073 **** 2021-09-16T19:28:12.663 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-16T19:28:12.697 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:28:12 +0000 (0:00:00.041) 0:07:46.115 **** 2021-09-16T19:28:15.205 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-09-16T19:28:15.207 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-b9y_xa6k-build\nRequirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nInstalling collected packages: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-b9y_xa6k-build", "Requirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Installing collected packages: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0:", " Successfully uninstalled registries-conf-ctl-1.0", " Running setup.py in 2021-09-16T19:28:15.209 INFO:teuthology.task.ansible.out:stall for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2021-09-16T19:28:15.230 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:28:15 +0000 (0:00:02.531) 0:07:48.646 **** 2021-09-16T19:28:15.257 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-qq748qb_-build\nRequirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nInstalling collected packages: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-qq748qb_-build", "Requirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Installing collected packages: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0:", " Successfully uninstalled registries-conf-ctl-1.0", " Running setup.py in 2021-09-16T19:28:15.258 INFO:teuthology.task.ansible.out:stall for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2021-09-16T19:28:15.278 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:28:15 +0000 (0:00:00.050) 0:07:48.696 **** 2021-09-16T19:28:15.285 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-09-16T19:28:15.286 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:28:15.311 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:28:15 +0000 (0:00:00.032) 0:07:48.729 **** 2021-09-16T19:28:15.317 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:28:15.340 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:28:15 +0000 (0:00:00.029) 0:07:48.758 **** 2021-09-16T19:28:15.347 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-09-16T19:28:15.348 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:28:15.370 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:28:15 +0000 (0:00:00.029) 0:07:48.788 **** 2021-09-16T19:28:15.376 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:28:15.401 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:28:15 +0000 (0:00:00.030) 0:07:48.818 **** 2021-09-16T19:28:15.718 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-09-16T19:28:15.720 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.159644", "end": "2021-09-16 19:28:15.700049", "rc": 0, "start": "2021-09-16 19:28:15.540405", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:28:15.742 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:28:15 +0000 (0:00:00.341) 0:07:49.160 **** 2021-09-16T19:28:15.760 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.163848", "end": "2021-09-16 19:28:15.732748", "rc": 0, "start": "2021-09-16 19:28:15.568900", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-16T19:28:15.782 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:28:15 +0000 (0:00:00.040) 0:07:49.200 **** 2021-09-16T19:28:15.789 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-09-16T19:28:15.790 INFO:teuthology.task.ansible.out:skipping: [smithi118.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:28:15.813 INFO:teuthology.task.ansible.out:skipping: [smithi109.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-16T19:28:15.888 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-09-16T19:28:15.890 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-09-16T19:28:15.891 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-09-16T19:28:15.893 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-09-16T19:28:15.904 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-09-16T19:28:15.906 INFO:teuthology.task.ansible.out:Thursday 16 September 2021 19:28:15 +0000 (0:00:00.122) 0:07:49.322 **** 2021-09-16T19:28:16.550 INFO:teuthology.task.ansible.out:ok: [smithi109.front.sepia.ceph.com] 2021-09-16T19:28:16.569 INFO:teuthology.task.ansible.out:ok: [smithi118.front.sepia.ceph.com] 2021-09-16T19:28:16.587 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** Thursday 16 September 2021 19:28:16 +0000 (0:00:00.683) 0:07:50.005 **** 2021-09-16T19:28:16.779 INFO:teuthology.task.ansible.out:changed: [smithi109.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-16T19:28:16.811 INFO:teuthology.task.ansible.out:changed: [smithi118.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-16T19:28:16.813 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi109.front.sepia.ceph.com : ok=142 changed=42 unreachable=0 failed=0 skipped=178 rescued=0 ignored=4 2021-09-16T19:28:16.815 INFO:teuthology.task.ansible.out:smithi118.front.sepia.ceph.com : ok=142 changed=42 unreachable=0 failed=0 skipped=178 rescued=0 ignored=4 Thursday 16 September 2021 19:28:16 +0000 (0:00:00.225) 0:07:50.231 **** =============================================================================== Create all admin users with sudo access. ------------------------------- 58.13s common : nrpe - Ensure SELinux policy is up to date -------------------- 52.33s users : Update authorized_keys using the keys repo --------------------- 35.23s testnode : Install packages -------------------------------------------- 31.82s testnode : Ensure ceph packages are not present. ----------------------- 30.00s testnode : Ensure ceph dependency packages are not present. ------------ 27.25s container-host : Install container packages ---------------------------- 19.82s Remove revoked users --------------------------------------------------- 18.85s common : Register with subscription-manager. --------------------------- 17.25s common : nrpe - Remove SELinux policy package -------------------------- 15.45s common : nrpe - Load SELinux policy package ---------------------------- 15.19s common : Install Kerberos Packages (RedHat) ---------------------------- 13.96s container-host : Ensure git is installed ------------------------------- 13.71s testnode : Install epel packages --------------------------------------- 13.48s testnode : Zap all non-root disks --------------------------------------- 7.63s common : Enable necessary rhsm repos ------------------------------------ 3.36s common : nrpe - Install SELinux dependencies ---------------------------- 3.02s testnode : Make sure rpm dependencies are installed --------------------- 2.95s common : Install nrpe package and dependencies (RHEL/CentOS) ------------ 2.95s common : Make sure smartmontools is installed --------------------------- 2.83s 2021-09-16T19:28:17.205 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi109.front.sepia.ceph.com'), Remote(name='ubuntu@smithi118.front.sepia.ceph.com')] 2021-09-16T19:28:17.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi109.front.sepia.ceph.com' 2021-09-16T19:28:17.208 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi109.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-16T19:28:17.276 DEBUG:teuthology.orchestra.run.smithi109:> true 2021-09-16T19:28:17.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2021-09-16T19:28:17.346 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-16T19:28:17.416 DEBUG:teuthology.orchestra.run.smithi118:> true 2021-09-16T19:28:17.485 INFO:teuthology.run_tasks:Running task clock... 2021-09-16T19:28:17.496 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-09-16T19:28:17.497 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-09-16T19:28:17.499 DEBUG:teuthology.orchestra.run.smithi109:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-09-16T19:28:17.528 INFO:teuthology.orchestra.run.smithi109.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-09-16T19:28:17.543 INFO:teuthology.orchestra.run.smithi109.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-09-16T19:28:17.580 INFO:teuthology.orchestra.run.smithi109.stderr:sudo: ntpd: command not found 2021-09-16T19:28:17.589 INFO:teuthology.orchestra.run.smithi109.stdout:506 Cannot talk to daemon 2021-09-16T19:28:17.604 INFO:teuthology.orchestra.run.smithi109.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-09-16T19:28:17.618 INFO:teuthology.orchestra.run.smithi109.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-09-16T19:28:17.684 INFO:teuthology.orchestra.run.smithi109.stderr:bash: ntpq: command not found 2021-09-16T19:28:17.686 INFO:teuthology.orchestra.run.smithi109.stdout:210 Number of sources = 4 2021-09-16T19:28:17.686 INFO:teuthology.orchestra.run.smithi109.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-09-16T19:28:17.687 INFO:teuthology.orchestra.run.smithi109.stdout:=============================================================================== 2021-09-16T19:28:17.687 INFO:teuthology.orchestra.run.smithi109.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-16T19:28:17.687 INFO:teuthology.orchestra.run.smithi109.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-16T19:28:17.687 INFO:teuthology.orchestra.run.smithi109.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-16T19:28:17.688 INFO:teuthology.orchestra.run.smithi109.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-16T19:28:17.689 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-09-16T19:28:17.689 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-09-16T19:28:17.721 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-09-16T19:28:17.736 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-09-16T19:28:17.768 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: ntpd: command not found 2021-09-16T19:28:17.778 INFO:teuthology.orchestra.run.smithi118.stdout:506 Cannot talk to daemon 2021-09-16T19:28:17.793 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-09-16T19:28:17.807 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-09-16T19:28:17.876 INFO:teuthology.orchestra.run.smithi118.stderr:bash: ntpq: command not found 2021-09-16T19:28:17.879 INFO:teuthology.orchestra.run.smithi118.stdout:210 Number of sources = 4 2021-09-16T19:28:17.880 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-09-16T19:28:17.880 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2021-09-16T19:28:17.880 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-16T19:28:17.880 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-16T19:28:17.880 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-16T19:28:17.881 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-16T19:28:17.881 INFO:teuthology.run_tasks:Running task install... 2021-09-16T19:28:17.896 DEBUG:teuthology.task.install:project ceph 2021-09-16T19:28:17.896 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'bc749ef43279601f3e9305d4ad8723d440abaa66'}} 2021-09-16T19:28:17.897 DEBUG:teuthology.task.install:config {'sha1': 'bc749ef43279601f3e9305d4ad8723d440abaa66'} 2021-09-16T19:28:17.897 INFO:teuthology.task.install:Using flavor: basic 2021-09-16T19:28:17.903 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-09-16T19:28:17.904 INFO:teuthology.task.install:extra packages: [] 2021-09-16T19:28:17.904 DEBUG:teuthology.orchestra.run.smithi109:> sudo lsb_release -is 2021-09-16T19:28:17.942 INFO:teuthology.orchestra.run.smithi109.stdout:RedHatEnterprise 2021-09-16T19:28:17.944 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-16T19:28:17.945 DEBUG:teuthology.orchestra.run.smithi118:> sudo lsb_release -is 2021-09-16T19:28:17.945 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc749ef43279601f3e9305d4ad8723d440abaa66', 'tag': None, 'wait_for_package': False} 2021-09-16T19:28:17.946 DEBUG:teuthology.orchestra.run.smithi109:> sudo lsb_release -is 2021-09-16T19:28:17.984 INFO:teuthology.orchestra.run.smithi118.stdout:RedHatEnterprise 2021-09-16T19:28:17.985 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-16T19:28:17.985 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bc749ef43279601f3e9305d4ad8723d440abaa66', 'tag': None, 'wait_for_package': False} 2021-09-16T19:28:17.986 DEBUG:teuthology.orchestra.run.smithi118:> sudo lsb_release -is 2021-09-16T19:28:18.022 INFO:teuthology.orchestra.run.smithi109.stdout:RedHatEnterprise 2021-09-16T19:28:18.024 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-16T19:28:18.024 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=bc749ef43279601f3e9305d4ad8723d440abaa66 2021-09-16T19:28:18.064 INFO:teuthology.orchestra.run.smithi118.stdout:RedHatEnterprise 2021-09-16T19:28:18.066 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-16T19:28:18.066 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=bc749ef43279601f3e9305d4ad8723d440abaa66 2021-09-16T19:28:18.174 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rgwadminops-fsid/bc749ef43279601f3e9305d4ad8723d440abaa66/centos/8/flavors/default/ 2021-09-16T19:28:18.174 INFO:teuthology.task.install.rpm:Package version is 17.0.0-7730.gbc749ef4 2021-09-16T19:28:18.212 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rgwadminops-fsid/bc749ef43279601f3e9305d4ad8723d440abaa66/centos/8/flavors/default/ 2021-09-16T19:28:18.213 INFO:teuthology.task.install.rpm:Package version is 17.0.0-7730.gbc749ef4 2021-09-16T19:28:18.299 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rgwadminops-fsid/bc749ef43279601f3e9305d4ad8723d440abaa66/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rgwadminops-fsid/bc749ef43279601f3e9305d4ad8723d440abaa66/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rgwadminops-fsid/bc749ef43279601f3e9305d4ad8723d440abaa66/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-09-16T19:28:18.300 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:28:18.300 DEBUG:teuthology.orchestra.run.smithi109:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-09-16T19:28:18.324 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-09-16T19:28:18.325 DEBUG:teuthology.orchestra.run.smithi109:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc749ef43279601f3e9305d4ad8723d440abaa66/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-09-16T19:28:18.327 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rgwadminops-fsid/bc749ef43279601f3e9305d4ad8723d440abaa66/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rgwadminops-fsid/bc749ef43279601f3e9305d4ad8723d440abaa66/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rgwadminops-fsid/bc749ef43279601f3e9305d4ad8723d440abaa66/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-09-16T19:28:18.328 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:28:18.328 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-09-16T19:28:18.352 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-09-16T19:28:18.352 DEBUG:teuthology.orchestra.run.smithi118:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bc749ef43279601f3e9305d4ad8723d440abaa66/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-09-16T19:28:18.389 DEBUG:teuthology.orchestra.run.smithi109:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-09-16T19:28:18.416 DEBUG:teuthology.orchestra.run.smithi118:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-09-16T19:28:18.463 DEBUG:teuthology.orchestra.run.smithi109:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-09-16T19:28:18.489 DEBUG:teuthology.orchestra.run.smithi118:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-09-16T19:28:18.526 INFO:teuthology.orchestra.run.smithi109.stdout:check_obsoletes = 1 2021-09-16T19:28:18.528 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum clean all 2021-09-16T19:28:18.551 INFO:teuthology.orchestra.run.smithi118.stdout:check_obsoletes = 1 2021-09-16T19:28:18.552 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum clean all 2021-09-16T19:28:18.882 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:28:18.905 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:28:19.175 INFO:teuthology.orchestra.run.smithi109.stdout:62 files removed 2021-09-16T19:28:19.203 INFO:teuthology.orchestra.run.smithi118.stdout:62 files removed 2021-09-16T19:28:19.218 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install ceph-radosgw 2021-09-16T19:28:19.247 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-radosgw 2021-09-16T19:28:19.532 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:28:19.560 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:28:20.154 INFO:teuthology.orchestra.run.smithi109.stdout:ceph packages for x86_64 323 kB/s | 75 kB 00:00 2021-09-16T19:28:20.482 INFO:teuthology.orchestra.run.smithi118.stdout:ceph packages for x86_64 139 kB/s | 75 kB 00:00 2021-09-16T19:28:20.496 INFO:teuthology.orchestra.run.smithi109.stdout:ceph noarch packages 29 kB/s | 9.0 kB 00:00 2021-09-16T19:28:20.905 INFO:teuthology.orchestra.run.smithi109.stdout:ceph source packages 3.6 kB/s | 1.4 kB 00:00 2021-09-16T19:28:20.905 INFO:teuthology.orchestra.run.smithi118.stdout:ceph noarch packages 23 kB/s | 9.0 kB 00:00 2021-09-16T19:28:21.332 INFO:teuthology.orchestra.run.smithi118.stdout:ceph source packages 3.5 kB/s | 1.4 kB 00:00 2021-09-16T19:28:23.406 INFO:teuthology.orchestra.run.smithi109.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2021-09-16T19:28:23.425 INFO:teuthology.orchestra.run.smithi118.stdout:Copr repo for python3-asyncssh owned by ceph 1.7 kB/s | 3.5 kB 00:02 2021-09-16T19:28:24.906 INFO:teuthology.orchestra.run.smithi109.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 10 MB 00:01 2021-09-16T19:28:25.519 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 5.0 MB/s | 10 MB 00:02 2021-09-16T19:28:26.910 INFO:teuthology.orchestra.run.smithi109.stdout:lab-extras 4.1 MB/s | 24 kB 00:00 2021-09-16T19:28:27.216 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 128 MB/s | 32 MB 00:00 2021-09-16T19:28:27.542 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2021-09-16T19:28:27.872 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 118 MB/s | 32 MB 00:00 2021-09-16T19:28:32.140 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 142 MB/s | 37 MB 00:00 2021-09-16T19:28:32.818 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 133 MB/s | 37 MB 00:00 2021-09-16T19:28:40.352 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:28:40.354 INFO:teuthology.orchestra.run.smithi109.stdout:==================================================================================================== 2021-09-16T19:28:40.354 INFO:teuthology.orchestra.run.smithi109.stdout: Package Arch Version Repository Size 2021-09-16T19:28:40.355 INFO:teuthology.orchestra.run.smithi109.stdout:==================================================================================================== 2021-09-16T19:28:40.355 INFO:teuthology.orchestra.run.smithi109.stdout:Installing: 2021-09-16T19:28:40.355 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-radosgw x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 8.4 M 2021-09-16T19:28:40.355 INFO:teuthology.orchestra.run.smithi109.stdout:Upgrading: 2021-09-16T19:28:40.355 INFO:teuthology.orchestra.run.smithi109.stdout: librados2 x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 3.4 M 2021-09-16T19:28:40.356 INFO:teuthology.orchestra.run.smithi109.stdout: librbd1 x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 3.9 M 2021-09-16T19:28:40.356 INFO:teuthology.orchestra.run.smithi109.stdout:Installing dependencies: 2021-09-16T19:28:40.356 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-base x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 5.2 M 2021-09-16T19:28:40.356 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-common x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 20 M 2021-09-16T19:28:40.357 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-selinux x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 24 k 2021-09-16T19:28:40.357 INFO:teuthology.orchestra.run.smithi109.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-09-16T19:28:40.357 INFO:teuthology.orchestra.run.smithi109.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-09-16T19:28:40.357 INFO:teuthology.orchestra.run.smithi109.stdout: libcephfs2 x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 782 k 2021-09-16T19:28:40.357 INFO:teuthology.orchestra.run.smithi109.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-09-16T19:28:40.358 INFO:teuthology.orchestra.run.smithi109.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2021-09-16T19:28:40.358 INFO:teuthology.orchestra.run.smithi109.stdout: libradosstriper1 x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 481 k 2021-09-16T19:28:40.358 INFO:teuthology.orchestra.run.smithi109.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2021-09-16T19:28:40.358 INFO:teuthology.orchestra.run.smithi109.stdout: librgw2 x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 3.7 M 2021-09-16T19:28:40.359 INFO:teuthology.orchestra.run.smithi109.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-09-16T19:28:40.359 INFO:teuthology.orchestra.run.smithi109.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2021-09-16T19:28:40.359 INFO:teuthology.orchestra.run.smithi109.stdout: python3-ceph-argparse x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 45 k 2021-09-16T19:28:40.359 INFO:teuthology.orchestra.run.smithi109.stdout: python3-ceph-common x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 78 k 2021-09-16T19:28:40.360 INFO:teuthology.orchestra.run.smithi109.stdout: python3-cephfs x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 178 k 2021-09-16T19:28:40.360 INFO:teuthology.orchestra.run.smithi109.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2021-09-16T19:28:40.360 INFO:teuthology.orchestra.run.smithi109.stdout: python3-rados x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 364 k 2021-09-16T19:28:40.360 INFO:teuthology.orchestra.run.smithi109.stdout: python3-rbd x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 346 k 2021-09-16T19:28:40.361 INFO:teuthology.orchestra.run.smithi109.stdout: python3-rgw x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 110 k 2021-09-16T19:28:40.361 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:28:40.361 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction Summary 2021-09-16T19:28:40.361 INFO:teuthology.orchestra.run.smithi109.stdout:==================================================================================================== 2021-09-16T19:28:40.361 INFO:teuthology.orchestra.run.smithi109.stdout:Install 21 Packages 2021-09-16T19:28:40.362 INFO:teuthology.orchestra.run.smithi109.stdout:Upgrade 2 Packages 2021-09-16T19:28:40.362 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:28:40.363 INFO:teuthology.orchestra.run.smithi109.stdout:Total download size: 48 M 2021-09-16T19:28:40.363 INFO:teuthology.orchestra.run.smithi109.stdout:Downloading Packages: 2021-09-16T19:28:40.903 INFO:teuthology.orchestra.run.smithi109.stdout:(1/23): ceph-base-17.0.0-7730.gbc749ef4.el8.x86 9.6 MB/s | 5.2 MB 00:00 2021-09-16T19:28:40.930 INFO:teuthology.orchestra.run.smithi109.stdout:(2/23): ceph-selinux-17.0.0-7730.gbc749ef4.el8. 903 kB/s | 24 kB 00:00 2021-09-16T19:28:40.989 INFO:teuthology.orchestra.run.smithi109.stdout:(3/23): libcephfs2-17.0.0-7730.gbc749ef4.el8.x8 13 MB/s | 782 kB 00:00 2021-09-16T19:28:41.037 INFO:teuthology.orchestra.run.smithi109.stdout:(4/23): libradosstriper1-17.0.0-7730.gbc749ef4. 9.8 MB/s | 481 kB 00:00 2021-09-16T19:28:41.196 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:28:41.198 INFO:teuthology.orchestra.run.smithi118.stdout:==================================================================================================== 2021-09-16T19:28:41.199 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2021-09-16T19:28:41.199 INFO:teuthology.orchestra.run.smithi118.stdout:==================================================================================================== 2021-09-16T19:28:41.199 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2021-09-16T19:28:41.199 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 8.4 M 2021-09-16T19:28:41.200 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrading: 2021-09-16T19:28:41.200 INFO:teuthology.orchestra.run.smithi118.stdout: librados2 x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 3.4 M 2021-09-16T19:28:41.200 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1 x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 3.9 M 2021-09-16T19:28:41.200 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2021-09-16T19:28:41.200 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 5.2 M 2021-09-16T19:28:41.201 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 20 M 2021-09-16T19:28:41.201 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 24 k 2021-09-16T19:28:41.201 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-09-16T19:28:41.201 INFO:teuthology.orchestra.run.smithi118.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-09-16T19:28:41.201 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2 x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 782 k 2021-09-16T19:28:41.202 INFO:teuthology.orchestra.run.smithi118.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-09-16T19:28:41.202 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2021-09-16T19:28:41.202 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1 x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 481 k 2021-09-16T19:28:41.202 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2021-09-16T19:28:41.202 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2 x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 3.7 M 2021-09-16T19:28:41.203 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-09-16T19:28:41.203 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2021-09-16T19:28:41.203 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 45 k 2021-09-16T19:28:41.203 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 78 k 2021-09-16T19:28:41.204 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 178 k 2021-09-16T19:28:41.204 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2021-09-16T19:28:41.204 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 364 k 2021-09-16T19:28:41.204 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 346 k 2021-09-16T19:28:41.204 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 110 k 2021-09-16T19:28:41.205 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:28:41.205 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2021-09-16T19:28:41.205 INFO:teuthology.orchestra.run.smithi118.stdout:==================================================================================================== 2021-09-16T19:28:41.205 INFO:teuthology.orchestra.run.smithi118.stdout:Install 21 Packages 2021-09-16T19:28:41.205 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrade 2 Packages 2021-09-16T19:28:41.206 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:28:41.206 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 48 M 2021-09-16T19:28:41.207 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2021-09-16T19:28:41.333 INFO:teuthology.orchestra.run.smithi109.stdout:(5/23): librgw2-17.0.0-7730.gbc749ef4.el8.x86_6 13 MB/s | 3.7 MB 00:00 2021-09-16T19:28:41.356 INFO:teuthology.orchestra.run.smithi109.stdout:(6/23): ceph-radosgw-17.0.0-7730.gbc749ef4.el8. 8.4 MB/s | 8.4 MB 00:00 2021-09-16T19:28:41.356 INFO:teuthology.orchestra.run.smithi109.stdout:(7/23): python3-ceph-argparse-17.0.0-7730.gbc74 1.9 MB/s | 45 kB 00:00 2021-09-16T19:28:41.381 INFO:teuthology.orchestra.run.smithi109.stdout:(8/23): python3-ceph-common-17.0.0-7730.gbc749e 3.0 MB/s | 78 kB 00:00 2021-09-16T19:28:41.387 INFO:teuthology.orchestra.run.smithi109.stdout:(9/23): python3-cephfs-17.0.0-7730.gbc749ef4.el 5.6 MB/s | 178 kB 00:00 2021-09-16T19:28:41.416 INFO:teuthology.orchestra.run.smithi109.stdout:(10/23): python3-rados-17.0.0-7730.gbc749ef4.el 10 MB/s | 364 kB 00:00 2021-09-16T19:28:41.429 INFO:teuthology.orchestra.run.smithi109.stdout:(11/23): python3-rbd-17.0.0-7730.gbc749ef4.el8. 8.3 MB/s | 346 kB 00:00 2021-09-16T19:28:41.451 INFO:teuthology.orchestra.run.smithi109.stdout:(12/23): python3-rgw-17.0.0-7730.gbc749ef4.el8. 3.1 MB/s | 110 kB 00:00 2021-09-16T19:28:41.576 INFO:teuthology.orchestra.run.smithi109.stdout:(13/23): leveldb-1.22-1.el8.x86_64.rpm 1.4 MB/s | 181 kB 00:00 2021-09-16T19:28:41.599 INFO:teuthology.orchestra.run.smithi109.stdout:(14/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2021-09-16T19:28:41.610 INFO:teuthology.orchestra.run.smithi109.stdout:(15/23): liboath-2.6.2-3.el8.x86_64.rpm 1.7 MB/s | 59 kB 00:00 2021-09-16T19:28:41.634 INFO:teuthology.orchestra.run.smithi109.stdout:(16/23): libunwind-1.3.1-3.el8.x86_64.rpm 2.1 MB/s | 75 kB 00:00 2021-09-16T19:28:41.645 INFO:teuthology.orchestra.run.smithi109.stdout:(17/23): librdkafka-0.11.4-1.el8.x86_64.rpm 10 MB/s | 353 kB 00:00 2021-09-16T19:28:41.669 INFO:teuthology.orchestra.run.smithi109.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.4 MB/s | 259 kB 00:00 2021-09-16T19:28:41.675 INFO:teuthology.orchestra.run.smithi109.stdout:(19/23): python3-prettytable-0.7.2-14.el8.noarc 1.4 MB/s | 44 kB 00:00 2021-09-16T19:28:41.688 INFO:teuthology.orchestra.run.smithi109.stdout:(20/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.4 MB/s | 47 kB 00:00 2021-09-16T19:28:41.992 INFO:teuthology.orchestra.run.smithi109.stdout:(21/23): librados2-17.0.0-7730.gbc749ef4.el8.x8 11 MB/s | 3.4 MB 00:00 2021-09-16T19:28:42.150 INFO:teuthology.orchestra.run.smithi109.stdout:(22/23): librbd1-17.0.0-7730.gbc749ef4.el8.x86_ 8.3 MB/s | 3.9 MB 00:00 2021-09-16T19:28:42.755 INFO:teuthology.orchestra.run.smithi118.stdout:(1/23): ceph-base-17.0.0-7730.gbc749ef4.el8.x86 3.4 MB/s | 5.2 MB 00:01 2021-09-16T19:28:42.802 INFO:teuthology.orchestra.run.smithi118.stdout:(2/23): ceph-selinux-17.0.0-7730.gbc749ef4.el8. 520 kB/s | 24 kB 00:00 2021-09-16T19:28:42.922 INFO:teuthology.orchestra.run.smithi118.stdout:(3/23): libcephfs2-17.0.0-7730.gbc749ef4.el8.x8 6.4 MB/s | 782 kB 00:00 2021-09-16T19:28:43.012 INFO:teuthology.orchestra.run.smithi118.stdout:(4/23): libradosstriper1-17.0.0-7730.gbc749ef4. 5.3 MB/s | 481 kB 00:00 2021-09-16T19:28:43.080 INFO:teuthology.orchestra.run.smithi109.stdout:(23/23): ceph-common-17.0.0-7730.gbc749ef4.el8. 7.2 MB/s | 20 MB 00:02 2021-09-16T19:28:43.084 INFO:teuthology.orchestra.run.smithi109.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:28:43.084 INFO:teuthology.orchestra.run.smithi109.stdout:Total 18 MB/s | 48 MB 00:02 2021-09-16T19:28:43.089 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction check 2021-09-16T19:28:43.118 INFO:teuthology.orchestra.run.smithi118.stdout:(5/23): ceph-radosgw-17.0.0-7730.gbc749ef4.el8. 4.4 MB/s | 8.4 MB 00:01 2021-09-16T19:28:43.137 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction check succeeded. 2021-09-16T19:28:43.138 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction test 2021-09-16T19:28:43.158 INFO:teuthology.orchestra.run.smithi118.stdout:(6/23): python3-ceph-argparse-17.0.0-7730.gbc74 1.1 MB/s | 45 kB 00:00 2021-09-16T19:28:43.191 INFO:teuthology.orchestra.run.smithi118.stdout:(7/23): python3-ceph-common-17.0.0-7730.gbc749e 2.3 MB/s | 78 kB 00:00 2021-09-16T19:28:43.221 INFO:teuthology.orchestra.run.smithi118.stdout:(8/23): python3-cephfs-17.0.0-7730.gbc749ef4.el 5.9 MB/s | 178 kB 00:00 2021-09-16T19:28:43.261 INFO:teuthology.orchestra.run.smithi118.stdout:(9/23): python3-rados-17.0.0-7730.gbc749ef4.el8 9.0 MB/s | 364 kB 00:00 2021-09-16T19:28:43.299 INFO:teuthology.orchestra.run.smithi118.stdout:(10/23): python3-rbd-17.0.0-7730.gbc749ef4.el8. 8.8 MB/s | 346 kB 00:00 2021-09-16T19:28:43.333 INFO:teuthology.orchestra.run.smithi118.stdout:(11/23): librgw2-17.0.0-7730.gbc749ef4.el8.x86_ 12 MB/s | 3.7 MB 00:00 2021-09-16T19:28:43.334 INFO:teuthology.orchestra.run.smithi118.stdout:(12/23): python3-rgw-17.0.0-7730.gbc749ef4.el8. 3.1 MB/s | 110 kB 00:00 2021-09-16T19:28:43.434 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction test succeeded. 2021-09-16T19:28:43.457 INFO:teuthology.orchestra.run.smithi118.stdout:(13/23): leveldb-1.22-1.el8.x86_64.rpm 1.4 MB/s | 181 kB 00:00 2021-09-16T19:28:43.474 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction 2021-09-16T19:28:43.480 INFO:teuthology.orchestra.run.smithi118.stdout:(14/23): gperftools-libs-2.7-9.el8.x86_64.rpm 2.0 MB/s | 306 kB 00:00 2021-09-16T19:28:43.482 INFO:teuthology.orchestra.run.smithi118.stdout:(15/23): liboath-2.6.2-3.el8.x86_64.rpm 2.3 MB/s | 59 kB 00:00 2021-09-16T19:28:43.506 INFO:teuthology.orchestra.run.smithi118.stdout:(16/23): libunwind-1.3.1-3.el8.x86_64.rpm 2.9 MB/s | 75 kB 00:00 2021-09-16T19:28:43.516 INFO:teuthology.orchestra.run.smithi118.stdout:(17/23): librdkafka-0.11.4-1.el8.x86_64.rpm 10 MB/s | 353 kB 00:00 2021-09-16T19:28:43.540 INFO:teuthology.orchestra.run.smithi118.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.4 MB/s | 259 kB 00:00 2021-09-16T19:28:43.547 INFO:teuthology.orchestra.run.smithi118.stdout:(19/23): python3-prettytable-0.7.2-14.el8.noarc 1.6 MB/s | 44 kB 00:00 2021-09-16T19:28:43.559 INFO:teuthology.orchestra.run.smithi118.stdout:(20/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.4 MB/s | 47 kB 00:00 2021-09-16T19:28:43.824 INFO:teuthology.orchestra.run.smithi118.stdout:(21/23): librados2-17.0.0-7730.gbc749ef4.el8.x8 12 MB/s | 3.4 MB 00:00 2021-09-16T19:28:43.880 INFO:teuthology.orchestra.run.smithi118.stdout:(22/23): librbd1-17.0.0-7730.gbc749ef4.el8.x86_ 12 MB/s | 3.9 MB 00:00 2021-09-16T19:28:44.614 INFO:teuthology.orchestra.run.smithi118.stdout:(23/23): ceph-common-17.0.0-7730.gbc749ef4.el8. 5.7 MB/s | 20 MB 00:03 2021-09-16T19:28:44.618 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:28:44.618 INFO:teuthology.orchestra.run.smithi118.stdout:Total 14 MB/s | 48 MB 00:03 2021-09-16T19:28:44.623 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2021-09-16T19:28:44.669 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2021-09-16T19:28:44.670 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2021-09-16T19:28:44.939 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2021-09-16T19:28:44.944 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2021-09-16T19:28:44.994 INFO:teuthology.orchestra.run.smithi109.stdout: Preparing : 1/1 2021-09-16T19:28:45.166 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2021-09-16T19:28:45.550 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2021-09-16T19:28:45.783 INFO:teuthology.orchestra.run.smithi109.stdout: Upgrading : librados2-2:17.0.0-7730.gbc749ef4.el8.x86_64 2/25 2021-09-16T19:28:45.881 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: librados2-2:17.0.0-7730.gbc749ef4.el8.x86_64 2/25 2021-09-16T19:28:45.956 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2021-09-16T19:28:46.046 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-rados-2:17.0.0-7730.gbc749ef4.el8.x86_64 3/25 2021-09-16T19:28:46.149 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2021-09-16T19:28:46.306 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2021-09-16T19:28:46.524 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : libcephfs2-2:17.0.0-7730.gbc749ef4.el8.x86_64 5/25 2021-09-16T19:28:46.532 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2021-09-16T19:28:46.733 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librados2-2:17.0.0-7730.gbc749ef4.el8.x86_64 2/25 2021-09-16T19:28:46.846 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:17.0.0-7730.gbc749ef4.el8.x86_64 2/25 2021-09-16T19:28:46.956 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: libcephfs2-2:17.0.0-7730.gbc749ef4.el8.x86_64 5/25 2021-09-16T19:28:47.055 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rados-2:17.0.0-7730.gbc749ef4.el8.x86_64 3/25 2021-09-16T19:28:47.208 INFO:teuthology.orchestra.run.smithi109.stdout: Upgrading : librbd1-2:17.0.0-7730.gbc749ef4.el8.x86_64 6/25 2021-09-16T19:28:47.271 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: librbd1-2:17.0.0-7730.gbc749ef4.el8.x86_64 6/25 2021-09-16T19:28:47.289 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2021-09-16T19:28:47.456 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2021-09-16T19:28:47.491 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs2-2:17.0.0-7730.gbc749ef4.el8.x86_64 5/25 2021-09-16T19:28:47.607 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-16T19:28:47.944 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs2-2:17.0.0-7730.gbc749ef4.el8.x86_64 5/25 2021-09-16T19:28:47.989 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-16T19:28:48.149 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : librgw2-2:17.0.0-7730.gbc749ef4.el8.x86_64 9/25 2021-09-16T19:28:48.166 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librbd1-2:17.0.0-7730.gbc749ef4.el8.x86_64 6/25 2021-09-16T19:28:48.212 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: librgw2-2:17.0.0-7730.gbc749ef4.el8.x86_64 9/25 2021-09-16T19:28:48.236 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:17.0.0-7730.gbc749ef4.el8.x86_64 6/25 2021-09-16T19:28:48.422 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2021-09-16T19:28:48.456 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2021-09-16T19:28:48.600 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-ceph-argparse-2:17.0.0-7730.gbc749ef4.el8. 11/25 2021-09-16T19:28:48.624 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-16T19:28:48.773 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-cephfs-2:17.0.0-7730.gbc749ef4.el8.x86_64 12/25 2021-09-16T19:28:48.974 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-rgw-2:17.0.0-7730.gbc749ef4.el8.x86_64 13/25 2021-09-16T19:28:49.009 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-16T19:28:49.183 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librgw2-2:17.0.0-7730.gbc749ef4.el8.x86_64 9/25 2021-09-16T19:28:49.209 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-rbd-2:17.0.0-7730.gbc749ef4.el8.x86_64 14/25 2021-09-16T19:28:49.260 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librgw2-2:17.0.0-7730.gbc749ef4.el8.x86_64 9/25 2021-09-16T19:28:49.374 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : libradosstriper1-2:17.0.0-7730.gbc749ef4.el8.x86_6 15/25 2021-09-16T19:28:49.432 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: libradosstriper1-2:17.0.0-7730.gbc749ef4.el8.x86_6 15/25 2021-09-16T19:28:49.438 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2021-09-16T19:28:49.566 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-argparse-2:17.0.0-7730.gbc749ef4.el8. 11/25 2021-09-16T19:28:49.617 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2021-09-16T19:28:49.769 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cephfs-2:17.0.0-7730.gbc749ef4.el8.x86_64 12/25 2021-09-16T19:28:49.807 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2021-09-16T19:28:49.990 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rgw-2:17.0.0-7730.gbc749ef4.el8.x86_64 13/25 2021-09-16T19:28:50.035 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2021-09-16T19:28:50.167 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rbd-2:17.0.0-7730.gbc749ef4.el8.x86_64 14/25 2021-09-16T19:28:50.233 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-ceph-common-2:17.0.0-7730.gbc749ef4.el8.x8 19/25 2021-09-16T19:28:50.316 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libradosstriper1-2:17.0.0-7730.gbc749ef4.el8.x86_6 15/25 2021-09-16T19:28:50.389 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libradosstriper1-2:17.0.0-7730.gbc749ef4.el8.x86_6 15/25 2021-09-16T19:28:50.542 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2021-09-16T19:28:50.773 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2021-09-16T19:28:51.003 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2021-09-16T19:28:51.174 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-common-2:17.0.0-7730.gbc749ef4.el8.x8 19/25 2021-09-16T19:28:54.337 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: ceph-common-2:17.0.0-7730.gbc749ef4.el8.x86_64 20/25 2021-09-16T19:28:54.557 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-common-2:17.0.0-7730.gbc749ef4.el8.x86_64 20/25 2021-09-16T19:28:55.123 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: ceph-common-2:17.0.0-7730.gbc749ef4.el8.x86_64 20/25 2021-09-16T19:28:55.398 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-base-2:17.0.0-7730.gbc749ef4.el8.x86_64 21/25 2021-09-16T19:28:55.399 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:17.0.0-7730.gbc749ef4.el8.x86_64 20/25 2021-09-16T19:28:55.503 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: ceph-base-2:17.0.0-7730.gbc749ef4.el8.x86_64 21/25 2021-09-16T19:28:55.607 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-common-2:17.0.0-7730.gbc749ef4.el8.x86_64 20/25 2021-09-16T19:28:55.656 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-selinux-2:17.0.0-7730.gbc749ef4.el8.x86_64 22/25 2021-09-16T19:28:56.172 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:17.0.0-7730.gbc749ef4.el8.x86_64 20/25 2021-09-16T19:28:56.403 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-base-2:17.0.0-7730.gbc749ef4.el8.x86_64 21/25 2021-09-16T19:28:56.516 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-base-2:17.0.0-7730.gbc749ef4.el8.x86_64 21/25 2021-09-16T19:28:56.656 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-selinux-2:17.0.0-7730.gbc749ef4.el8.x86_64 22/25 2021-09-16T19:29:13.491 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: ceph-selinux-2:17.0.0-7730.gbc749ef4.el8.x86_64 22/25 2021-09-16T19:29:13.492 INFO:teuthology.orchestra.run.smithi109.stdout:skipping the directory /sys 2021-09-16T19:29:13.492 INFO:teuthology.orchestra.run.smithi109.stdout:skipping the directory /proc 2021-09-16T19:29:13.492 INFO:teuthology.orchestra.run.smithi109.stdout:skipping the directory /dev 2021-09-16T19:29:13.492 INFO:teuthology.orchestra.run.smithi109.stdout:skipping the directory /run 2021-09-16T19:29:13.493 INFO:teuthology.orchestra.run.smithi109.stdout:skipping the directory /mnt 2021-09-16T19:29:13.493 INFO:teuthology.orchestra.run.smithi109.stdout:skipping the directory /var/tmp 2021-09-16T19:29:13.493 INFO:teuthology.orchestra.run.smithi109.stdout:skipping the directory /home 2021-09-16T19:29:13.493 INFO:teuthology.orchestra.run.smithi109.stdout:skipping the directory /tmp 2021-09-16T19:29:13.493 INFO:teuthology.orchestra.run.smithi109.stdout:skipping the directory /dev 2021-09-16T19:29:13.494 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:29:14.490 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-selinux-2:17.0.0-7730.gbc749ef4.el8.x86_64 22/25 2021-09-16T19:29:14.491 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /sys 2021-09-16T19:29:14.491 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /proc 2021-09-16T19:29:14.491 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /dev 2021-09-16T19:29:14.491 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /run 2021-09-16T19:29:14.491 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /mnt 2021-09-16T19:29:14.492 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /var/tmp 2021-09-16T19:29:14.492 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /home 2021-09-16T19:29:14.492 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /tmp 2021-09-16T19:29:14.492 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /dev 2021-09-16T19:29:14.493 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:29:14.579 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-radosgw-2:17.0.0-7730.gbc749ef4.el8.x86_64 23/25 2021-09-16T19:29:14.658 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-7730.gbc749ef4.el8.x86_64 23/25 2021-09-16T19:29:14.658 INFO:teuthology.orchestra.run.smithi109.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-16T19:29:14.906 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-16T19:29:14.907 INFO:teuthology.orchestra.run.smithi109.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-16T19:29:15.558 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-radosgw-2:17.0.0-7730.gbc749ef4.el8.x86_64 23/25 2021-09-16T19:29:15.622 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-7730.gbc749ef4.el8.x86_64 23/25 2021-09-16T19:29:15.622 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-16T19:29:15.705 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-16T19:29:15.705 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-base-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/25 2021-09-16T19:29:15.705 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-common-2:17.0.0-7730.gbc749ef4.el8.x86_64 2/25 2021-09-16T19:29:15.705 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-radosgw-2:17.0.0-7730.gbc749ef4.el8.x86_64 3/25 2021-09-16T19:29:15.706 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-selinux-2:17.0.0-7730.gbc749ef4.el8.x86_64 4/25 2021-09-16T19:29:15.706 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : libcephfs2-2:17.0.0-7730.gbc749ef4.el8.x86_64 5/25 2021-09-16T19:29:15.706 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : libradosstriper1-2:17.0.0-7730.gbc749ef4.el8.x86_6 6/25 2021-09-16T19:29:15.706 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : librgw2-2:17.0.0-7730.gbc749ef4.el8.x86_64 7/25 2021-09-16T19:29:15.706 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-ceph-argparse-2:17.0.0-7730.gbc749ef4.el8. 8/25 2021-09-16T19:29:15.707 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-ceph-common-2:17.0.0-7730.gbc749ef4.el8.x8 9/25 2021-09-16T19:29:15.707 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-cephfs-2:17.0.0-7730.gbc749ef4.el8.x86_64 10/25 2021-09-16T19:29:15.707 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-rados-2:17.0.0-7730.gbc749ef4.el8.x86_64 11/25 2021-09-16T19:29:15.707 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-rbd-2:17.0.0-7730.gbc749ef4.el8.x86_64 12/25 2021-09-16T19:29:15.708 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-rgw-2:17.0.0-7730.gbc749ef4.el8.x86_64 13/25 2021-09-16T19:29:15.708 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2021-09-16T19:29:15.708 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : leveldb-1.22-1.el8.x86_64 15/25 2021-09-16T19:29:15.708 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 16/25 2021-09-16T19:29:15.708 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 17/25 2021-09-16T19:29:15.709 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 18/25 2021-09-16T19:29:15.709 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2021-09-16T19:29:15.709 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/25 2021-09-16T19:29:15.709 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 21/25 2021-09-16T19:29:15.709 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : librados2-2:17.0.0-7730.gbc749ef4.el8.x86_64 22/25 2021-09-16T19:29:15.710 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2021-09-16T19:29:15.710 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : librbd1-2:17.0.0-7730.gbc749ef4.el8.x86_64 24/25 2021-09-16T19:29:15.855 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-16T19:29:15.856 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-16T19:29:16.551 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2021-09-16T19:29:16.551 INFO:teuthology.orchestra.run.smithi109.stdout:Installed products updated. 2021-09-16T19:29:16.649 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-16T19:29:16.649 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-base-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/25 2021-09-16T19:29:16.650 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-common-2:17.0.0-7730.gbc749ef4.el8.x86_64 2/25 2021-09-16T19:29:16.650 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-radosgw-2:17.0.0-7730.gbc749ef4.el8.x86_64 3/25 2021-09-16T19:29:16.650 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-selinux-2:17.0.0-7730.gbc749ef4.el8.x86_64 4/25 2021-09-16T19:29:16.650 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs2-2:17.0.0-7730.gbc749ef4.el8.x86_64 5/25 2021-09-16T19:29:16.650 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libradosstriper1-2:17.0.0-7730.gbc749ef4.el8.x86_6 6/25 2021-09-16T19:29:16.651 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librgw2-2:17.0.0-7730.gbc749ef4.el8.x86_64 7/25 2021-09-16T19:29:16.651 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-argparse-2:17.0.0-7730.gbc749ef4.el8. 8/25 2021-09-16T19:29:16.651 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-common-2:17.0.0-7730.gbc749ef4.el8.x8 9/25 2021-09-16T19:29:16.651 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cephfs-2:17.0.0-7730.gbc749ef4.el8.x86_64 10/25 2021-09-16T19:29:16.652 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:17.0.0-7730.gbc749ef4.el8.x86_64 11/25 2021-09-16T19:29:16.652 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rbd-2:17.0.0-7730.gbc749ef4.el8.x86_64 12/25 2021-09-16T19:29:16.652 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rgw-2:17.0.0-7730.gbc749ef4.el8.x86_64 13/25 2021-09-16T19:29:16.652 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2021-09-16T19:29:16.652 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : leveldb-1.22-1.el8.x86_64 15/25 2021-09-16T19:29:16.653 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 16/25 2021-09-16T19:29:16.653 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 17/25 2021-09-16T19:29:16.653 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 18/25 2021-09-16T19:29:16.653 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2021-09-16T19:29:16.653 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/25 2021-09-16T19:29:16.654 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 21/25 2021-09-16T19:29:16.654 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:17.0.0-7730.gbc749ef4.el8.x86_64 22/25 2021-09-16T19:29:16.654 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2021-09-16T19:29:16.654 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:17.0.0-7730.gbc749ef4.el8.x86_64 24/25 2021-09-16T19:29:17.408 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2021-09-16T19:29:17.409 INFO:teuthology.orchestra.run.smithi118.stdout:Installed products updated. 2021-09-16T19:29:27.226 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:04 ago on Thu 16 Sep 2021 07:29:21 PM UTC. 2021-09-16T19:29:28.238 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:29:22 PM UTC. 2021-09-16T19:29:29.730 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:29:29.731 INFO:teuthology.orchestra.run.smithi109.stdout:Upgraded: 2021-09-16T19:29:29.731 INFO:teuthology.orchestra.run.smithi109.stdout: librados2-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:29.731 INFO:teuthology.orchestra.run.smithi109.stdout: librbd1-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:29.731 INFO:teuthology.orchestra.run.smithi109.stdout:Installed: 2021-09-16T19:29:29.731 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-base-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:29.732 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-common-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:29.732 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-radosgw-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:29.732 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-selinux-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:29.732 INFO:teuthology.orchestra.run.smithi109.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-09-16T19:29:29.733 INFO:teuthology.orchestra.run.smithi109.stdout: leveldb-1.22-1.el8.x86_64 2021-09-16T19:29:29.733 INFO:teuthology.orchestra.run.smithi109.stdout: libcephfs2-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:29.733 INFO:teuthology.orchestra.run.smithi109.stdout: liboath-2.6.2-3.el8.x86_64 2021-09-16T19:29:29.733 INFO:teuthology.orchestra.run.smithi109.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-09-16T19:29:29.733 INFO:teuthology.orchestra.run.smithi109.stdout: libradosstriper1-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:29.734 INFO:teuthology.orchestra.run.smithi109.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-09-16T19:29:29.734 INFO:teuthology.orchestra.run.smithi109.stdout: librgw2-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:29.735 INFO:teuthology.orchestra.run.smithi109.stdout: libunwind-1.3.1-3.el8.x86_64 2021-09-16T19:29:29.735 INFO:teuthology.orchestra.run.smithi109.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-09-16T19:29:29.735 INFO:teuthology.orchestra.run.smithi109.stdout: python3-ceph-argparse-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:29.735 INFO:teuthology.orchestra.run.smithi109.stdout: python3-ceph-common-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:29.735 INFO:teuthology.orchestra.run.smithi109.stdout: python3-cephfs-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:29.736 INFO:teuthology.orchestra.run.smithi109.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-09-16T19:29:29.736 INFO:teuthology.orchestra.run.smithi109.stdout: python3-rados-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:29.736 INFO:teuthology.orchestra.run.smithi109.stdout: python3-rbd-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:29.736 INFO:teuthology.orchestra.run.smithi109.stdout: python3-rgw-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:29.736 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:29:29.737 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:29:29.855 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install ceph-test 2021-09-16T19:29:30.182 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:29:30.658 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-09-16T19:29:30.798 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:29:30.798 INFO:teuthology.orchestra.run.smithi118.stdout:Upgraded: 2021-09-16T19:29:30.799 INFO:teuthology.orchestra.run.smithi118.stdout: librados2-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:30.799 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:30.799 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2021-09-16T19:29:30.799 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:30.800 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:30.800 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:30.800 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:30.800 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-09-16T19:29:30.800 INFO:teuthology.orchestra.run.smithi118.stdout: leveldb-1.22-1.el8.x86_64 2021-09-16T19:29:30.801 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:30.801 INFO:teuthology.orchestra.run.smithi118.stdout: liboath-2.6.2-3.el8.x86_64 2021-09-16T19:29:30.801 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-09-16T19:29:30.801 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:30.803 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-09-16T19:29:30.804 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:30.804 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind-1.3.1-3.el8.x86_64 2021-09-16T19:29:30.804 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-09-16T19:29:30.804 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:30.805 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:30.805 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:30.805 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-09-16T19:29:30.805 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:30.805 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:30.806 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:29:30.806 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:29:30.806 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:29:30.942 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-test 2021-09-16T19:29:31.269 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:29:31.728 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-09-16T19:29:35.422 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-09-16T19:29:36.495 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 84 kB/s | 2.4 kB 00:00 2021-09-16T19:29:40.655 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:28:26 PM UTC. 2021-09-16T19:29:41.787 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:28:27 PM UTC. 2021-09-16T19:29:43.232 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:29:43.233 INFO:teuthology.orchestra.run.smithi109.stdout:============================================================================================ 2021-09-16T19:29:43.233 INFO:teuthology.orchestra.run.smithi109.stdout: Package Arch Version Repository Size 2021-09-16T19:29:43.233 INFO:teuthology.orchestra.run.smithi109.stdout:============================================================================================ 2021-09-16T19:29:43.233 INFO:teuthology.orchestra.run.smithi109.stdout:Installing: 2021-09-16T19:29:43.234 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-test x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 43 M 2021-09-16T19:29:43.234 INFO:teuthology.orchestra.run.smithi109.stdout:Installing dependencies: 2021-09-16T19:29:43.234 INFO:teuthology.orchestra.run.smithi109.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2021-09-16T19:29:43.234 INFO:teuthology.orchestra.run.smithi109.stdout: libcephsqlite x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 170 k 2021-09-16T19:29:43.235 INFO:teuthology.orchestra.run.smithi109.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2021-09-16T19:29:43.235 INFO:teuthology.orchestra.run.smithi109.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2021-09-16T19:29:43.235 INFO:teuthology.orchestra.run.smithi109.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-09-16T19:29:43.235 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:29:43.235 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction Summary 2021-09-16T19:29:43.236 INFO:teuthology.orchestra.run.smithi109.stdout:============================================================================================ 2021-09-16T19:29:43.236 INFO:teuthology.orchestra.run.smithi109.stdout:Install 6 Packages 2021-09-16T19:29:43.236 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:29:43.237 INFO:teuthology.orchestra.run.smithi109.stdout:Total download size: 44 M 2021-09-16T19:29:43.237 INFO:teuthology.orchestra.run.smithi109.stdout:Installed size: 206 M 2021-09-16T19:29:43.237 INFO:teuthology.orchestra.run.smithi109.stdout:Downloading Packages: 2021-09-16T19:29:43.340 INFO:teuthology.orchestra.run.smithi109.stdout:(1/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 660 kB/s | 69 kB 00:00 2021-09-16T19:29:43.371 INFO:teuthology.orchestra.run.smithi109.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 5.0 MB/s | 161 kB 00:00 2021-09-16T19:29:43.392 INFO:teuthology.orchestra.run.smithi109.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 8.8 MB/s | 187 kB 00:00 2021-09-16T19:29:43.409 INFO:teuthology.orchestra.run.smithi109.stdout:(4/6): libcephsqlite-17.0.0-7730.gbc749ef4.el8. 978 kB/s | 170 kB 00:00 2021-09-16T19:29:43.413 INFO:teuthology.orchestra.run.smithi109.stdout:(5/6): socat-1.7.3.3-2.el8.x86_64.rpm 14 MB/s | 302 kB 00:00 2021-09-16T19:29:44.341 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:29:44.342 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2021-09-16T19:29:44.342 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2021-09-16T19:29:44.342 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2021-09-16T19:29:44.343 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2021-09-16T19:29:44.343 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 43 M 2021-09-16T19:29:44.343 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2021-09-16T19:29:44.343 INFO:teuthology.orchestra.run.smithi118.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2021-09-16T19:29:44.343 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 170 k 2021-09-16T19:29:44.344 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2021-09-16T19:29:44.344 INFO:teuthology.orchestra.run.smithi118.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2021-09-16T19:29:44.344 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-09-16T19:29:44.344 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:29:44.345 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2021-09-16T19:29:44.345 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2021-09-16T19:29:44.345 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2021-09-16T19:29:44.345 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:29:44.346 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 44 M 2021-09-16T19:29:44.346 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 206 M 2021-09-16T19:29:44.346 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2021-09-16T19:29:44.458 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 607 kB/s | 69 kB 00:00 2021-09-16T19:29:44.489 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 5.2 MB/s | 161 kB 00:00 2021-09-16T19:29:44.509 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 9.0 MB/s | 187 kB 00:00 2021-09-16T19:29:44.531 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 14 MB/s | 302 kB 00:00 2021-09-16T19:29:44.809 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): libcephsqlite-17.0.0-7730.gbc749ef4.el8. 367 kB/s | 170 kB 00:00 2021-09-16T19:29:45.353 INFO:teuthology.orchestra.run.smithi109.stdout:(6/6): ceph-test-17.0.0-7730.gbc749ef4.el8.x86_ 20 MB/s | 43 MB 00:02 2021-09-16T19:29:45.356 INFO:teuthology.orchestra.run.smithi109.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:29:45.357 INFO:teuthology.orchestra.run.smithi109.stdout:Total 21 MB/s | 44 MB 00:02 2021-09-16T19:29:45.360 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction check 2021-09-16T19:29:45.382 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction check succeeded. 2021-09-16T19:29:45.382 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction test 2021-09-16T19:29:45.622 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction test succeeded. 2021-09-16T19:29:45.626 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction 2021-09-16T19:29:46.123 INFO:teuthology.orchestra.run.smithi109.stdout: Preparing : 1/1 2021-09-16T19:29:46.319 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2021-09-16T19:29:46.525 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2021-09-16T19:29:46.586 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2021-09-16T19:29:46.802 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2021-09-16T19:29:46.956 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 4/6 2021-09-16T19:29:47.100 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : libcephsqlite-2:17.0.0-7730.gbc749ef4.el8.x86_64 5/6 2021-09-16T19:29:47.605 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): ceph-test-17.0.0-7730.gbc749ef4.el8.x86_ 13 MB/s | 43 MB 00:03 2021-09-16T19:29:47.608 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:29:47.608 INFO:teuthology.orchestra.run.smithi118.stdout:Total 13 MB/s | 44 MB 00:03 2021-09-16T19:29:47.612 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2021-09-16T19:29:47.635 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2021-09-16T19:29:47.635 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2021-09-16T19:29:47.872 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2021-09-16T19:29:47.876 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2021-09-16T19:29:48.362 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2021-09-16T19:29:48.576 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2021-09-16T19:29:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2021-09-16T19:29:48.842 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2021-09-16T19:29:49.041 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2021-09-16T19:29:49.187 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 4/6 2021-09-16T19:29:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephsqlite-2:17.0.0-7730.gbc749ef4.el8.x86_64 5/6 2021-09-16T19:29:51.323 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: libcephsqlite-2:17.0.0-7730.gbc749ef4.el8.x86_64 5/6 2021-09-16T19:29:51.613 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-test-2:17.0.0-7730.gbc749ef4.el8.x86_64 6/6 2021-09-16T19:29:51.834 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: ceph-test-2:17.0.0-7730.gbc749ef4.el8.x86_64 6/6 2021-09-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-test-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/6 2021-09-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : libcephsqlite-2:17.0.0-7730.gbc749ef4.el8.x86_64 2/6 2021-09-16T19:29:51.835 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 3/6 2021-09-16T19:29:51.836 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2021-09-16T19:29:51.836 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2021-09-16T19:29:52.135 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2021-09-16T19:29:52.135 INFO:teuthology.orchestra.run.smithi109.stdout:Installed products updated. 2021-09-16T19:29:53.581 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephsqlite-2:17.0.0-7730.gbc749ef4.el8.x86_64 5/6 2021-09-16T19:29:53.869 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-test-2:17.0.0-7730.gbc749ef4.el8.x86_64 6/6 2021-09-16T19:29:54.089 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-test-2:17.0.0-7730.gbc749ef4.el8.x86_64 6/6 2021-09-16T19:29:54.090 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-test-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/6 2021-09-16T19:29:54.090 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephsqlite-2:17.0.0-7730.gbc749ef4.el8.x86_64 2/6 2021-09-16T19:29:54.090 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 3/6 2021-09-16T19:29:54.090 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2021-09-16T19:29:54.090 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2021-09-16T19:29:54.421 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2021-09-16T19:29:54.422 INFO:teuthology.orchestra.run.smithi118.stdout:Installed products updated. 2021-09-16T19:30:02.558 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:29:21 PM UTC. 2021-09-16T19:30:04.772 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:29:22 PM UTC. 2021-09-16T19:30:05.130 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:30:05.131 INFO:teuthology.orchestra.run.smithi109.stdout:Installed: 2021-09-16T19:30:05.131 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-test-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:30:05.131 INFO:teuthology.orchestra.run.smithi109.stdout: jq-1.5-12.el8.x86_64 2021-09-16T19:30:05.131 INFO:teuthology.orchestra.run.smithi109.stdout: libcephsqlite-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:30:05.132 INFO:teuthology.orchestra.run.smithi109.stdout: oniguruma-6.8.2-2.el8.x86_64 2021-09-16T19:30:05.132 INFO:teuthology.orchestra.run.smithi109.stdout: socat-1.7.3.3-2.el8.x86_64 2021-09-16T19:30:05.132 INFO:teuthology.orchestra.run.smithi109.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-09-16T19:30:05.132 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:30:05.133 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:30:05.261 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install ceph 2021-09-16T19:30:05.579 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:30:06.088 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-09-16T19:30:07.313 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:30:07.313 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2021-09-16T19:30:07.314 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:30:07.314 INFO:teuthology.orchestra.run.smithi118.stdout: jq-1.5-12.el8.x86_64 2021-09-16T19:30:07.314 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:30:07.314 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma-6.8.2-2.el8.x86_64 2021-09-16T19:30:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: socat-1.7.3.3-2.el8.x86_64 2021-09-16T19:30:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-09-16T19:30:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:30:07.315 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:30:07.448 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph 2021-09-16T19:30:07.768 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:30:08.269 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 91 kB/s | 2.8 kB 00:00 2021-09-16T19:30:10.886 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.4 kB 00:00 2021-09-16T19:30:13.063 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-09-16T19:30:16.093 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:04 ago on Thu 16 Sep 2021 07:28:26 PM UTC. 2021-09-16T19:30:18.322 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:04 ago on Thu 16 Sep 2021 07:28:27 PM UTC. 2021-09-16T19:30:18.590 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:30:18.593 INFO:teuthology.orchestra.run.smithi109.stdout:======================================================================================================= 2021-09-16T19:30:18.593 INFO:teuthology.orchestra.run.smithi109.stdout: Package Arch Version Repository Size 2021-09-16T19:30:18.594 INFO:teuthology.orchestra.run.smithi109.stdout:======================================================================================================= 2021-09-16T19:30:18.594 INFO:teuthology.orchestra.run.smithi109.stdout:Installing: 2021-09-16T19:30:18.594 INFO:teuthology.orchestra.run.smithi109.stdout: ceph x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 6.4 k 2021-09-16T19:30:18.594 INFO:teuthology.orchestra.run.smithi109.stdout:Installing dependencies: 2021-09-16T19:30:18.595 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-mds x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 2.3 M 2021-09-16T19:30:18.595 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-mgr x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 1.5 M 2021-09-16T19:30:18.595 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-mgr-modules-core noarch 2:17.0.0-7730.gbc749ef4.el8 ceph-noarch 214 k 2021-09-16T19:30:18.595 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-mon x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 3.9 M 2021-09-16T19:30:18.596 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-osd x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 17 M 2021-09-16T19:30:18.596 INFO:teuthology.orchestra.run.smithi109.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-09-16T19:30:18.596 INFO:teuthology.orchestra.run.smithi109.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-09-16T19:30:18.596 INFO:teuthology.orchestra.run.smithi109.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2021-09-16T19:30:18.597 INFO:teuthology.orchestra.run.smithi109.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-09-16T19:30:18.597 INFO:teuthology.orchestra.run.smithi109.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-09-16T19:30:18.597 INFO:teuthology.orchestra.run.smithi109.stdout: python3-cryptography x86_64 3.2.1-4.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2021-09-16T19:30:18.597 INFO:teuthology.orchestra.run.smithi109.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-09-16T19:30:18.597 INFO:teuthology.orchestra.run.smithi109.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-09-16T19:30:18.598 INFO:teuthology.orchestra.run.smithi109.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-09-16T19:30:18.598 INFO:teuthology.orchestra.run.smithi109.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2021-09-16T19:30:18.598 INFO:teuthology.orchestra.run.smithi109.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2021-09-16T19:30:18.598 INFO:teuthology.orchestra.run.smithi109.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-09-16T19:30:18.599 INFO:teuthology.orchestra.run.smithi109.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-09-16T19:30:18.599 INFO:teuthology.orchestra.run.smithi109.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-09-16T19:30:18.599 INFO:teuthology.orchestra.run.smithi109.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2021-09-16T19:30:18.599 INFO:teuthology.orchestra.run.smithi109.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2021-09-16T19:30:18.599 INFO:teuthology.orchestra.run.smithi109.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2021-09-16T19:30:18.600 INFO:teuthology.orchestra.run.smithi109.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-09-16T19:30:18.600 INFO:teuthology.orchestra.run.smithi109.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-09-16T19:30:18.600 INFO:teuthology.orchestra.run.smithi109.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-09-16T19:30:18.600 INFO:teuthology.orchestra.run.smithi109.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-09-16T19:30:18.601 INFO:teuthology.orchestra.run.smithi109.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-09-16T19:30:18.601 INFO:teuthology.orchestra.run.smithi109.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-09-16T19:30:18.601 INFO:teuthology.orchestra.run.smithi109.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-09-16T19:30:18.601 INFO:teuthology.orchestra.run.smithi109.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2021-09-16T19:30:18.601 INFO:teuthology.orchestra.run.smithi109.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-09-16T19:30:18.602 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:30:18.602 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction Summary 2021-09-16T19:30:18.602 INFO:teuthology.orchestra.run.smithi109.stdout:======================================================================================================= 2021-09-16T19:30:18.602 INFO:teuthology.orchestra.run.smithi109.stdout:Install 32 Packages 2021-09-16T19:30:18.603 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:30:18.603 INFO:teuthology.orchestra.run.smithi109.stdout:Total download size: 28 M 2021-09-16T19:30:18.604 INFO:teuthology.orchestra.run.smithi109.stdout:Installed size: 112 M 2021-09-16T19:30:18.604 INFO:teuthology.orchestra.run.smithi109.stdout:Downloading Packages: 2021-09-16T19:30:18.700 INFO:teuthology.orchestra.run.smithi109.stdout:(1/32): ceph-17.0.0-7730.gbc749ef4.el8.x86_64.r 64 kB/s | 6.4 kB 00:00 2021-09-16T19:30:18.961 INFO:teuthology.orchestra.run.smithi109.stdout:(2/32): ceph-mds-17.0.0-7730.gbc749ef4.el8.x86_ 6.2 MB/s | 2.3 MB 00:00 2021-09-16T19:30:18.965 INFO:teuthology.orchestra.run.smithi109.stdout:(3/32): ceph-mgr-17.0.0-7730.gbc749ef4.el8.x86_ 4.1 MB/s | 1.5 MB 00:00 2021-09-16T19:30:19.034 INFO:teuthology.orchestra.run.smithi109.stdout:(4/32): ceph-mgr-modules-core-17.0.0-7730.gbc74 3.0 MB/s | 214 kB 00:00 2021-09-16T19:30:19.112 INFO:teuthology.orchestra.run.smithi109.stdout:(5/32): ceph-mon-17.0.0-7730.gbc749ef4.el8.x86_ 9.4 MB/s | 3.9 MB 00:00 2021-09-16T19:30:19.144 INFO:teuthology.orchestra.run.smithi109.stdout:(6/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 398 kB/s | 44 kB 00:00 2021-09-16T19:30:19.226 INFO:teuthology.orchestra.run.smithi109.stdout:(7/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.1 MB/s | 173 kB 00:00 2021-09-16T19:30:19.237 INFO:teuthology.orchestra.run.smithi109.stdout:(8/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 1.5 MB/s | 185 kB 00:00 2021-09-16T19:30:19.261 INFO:teuthology.orchestra.run.smithi109.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 454 kB/s | 11 kB 00:00 2021-09-16T19:30:19.284 INFO:teuthology.orchestra.run.smithi109.stdout:(10/32): python3-cherrypy-18.4.0-1.el8.noarch.r 6.5 MB/s | 384 kB 00:00 2021-09-16T19:30:19.286 INFO:teuthology.orchestra.run.smithi109.stdout:(11/32): python3-jaraco-functools-2.0-4.el8.noa 716 kB/s | 18 kB 00:00 2021-09-16T19:30:19.311 INFO:teuthology.orchestra.run.smithi109.stdout:(12/32): python3-more-itertools-7.2.0-3.el8.noa 2.3 MB/s | 59 kB 00:00 2021-09-16T19:30:19.312 INFO:teuthology.orchestra.run.smithi109.stdout:(13/32): python3-logutils-0.3.5-11.el8.noarch.r 1.7 MB/s | 49 kB 00:00 2021-09-16T19:30:19.341 INFO:teuthology.orchestra.run.smithi109.stdout:(14/32): python3-portend-2.6-1.el8.noarch.rpm 550 kB/s | 16 kB 00:00 2021-09-16T19:30:19.361 INFO:teuthology.orchestra.run.smithi109.stdout:(15/32): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2021-09-16T19:30:19.369 INFO:teuthology.orchestra.run.smithi109.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 685 kB/s | 19 kB 00:00 2021-09-16T19:30:19.386 INFO:teuthology.orchestra.run.smithi109.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 988 kB/s | 24 kB 00:00 2021-09-16T19:30:19.396 INFO:teuthology.orchestra.run.smithi109.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 1.0 MB/s | 29 kB 00:00 2021-09-16T19:30:19.411 INFO:teuthology.orchestra.run.smithi109.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 1.1 MB/s | 27 kB 00:00 2021-09-16T19:30:19.429 INFO:teuthology.orchestra.run.smithi109.stdout:(20/32): python3-waitress-1.2.1-2.el8.1.noarch. 7.5 MB/s | 241 kB 00:00 2021-09-16T19:30:19.438 INFO:teuthology.orchestra.run.smithi109.stdout:(21/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 9.1 MB/s | 251 kB 00:00 2021-09-16T19:30:19.456 INFO:teuthology.orchestra.run.smithi109.stdout:(22/32): python3-webtest-2.0.33-1.el8.noarch.rp 3.0 MB/s | 85 kB 00:00 2021-09-16T19:30:19.465 INFO:teuthology.orchestra.run.smithi109.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 837 kB/s | 23 kB 00:00 2021-09-16T19:30:19.498 INFO:teuthology.orchestra.run.smithi109.stdout:(24/32): python3-mako-1.0.6-13.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2021-09-16T19:30:19.504 INFO:teuthology.orchestra.run.smithi109.stdout:(25/32): python3-pytz-2017.2-9.el8.noarch.rpm 1.4 MB/s | 54 kB 00:00 2021-09-16T19:30:19.523 INFO:teuthology.orchestra.run.smithi109.stdout:(26/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 4.1 MB/s | 103 kB 00:00 2021-09-16T19:30:19.532 INFO:teuthology.orchestra.run.smithi109.stdout:(27/32): python3-werkzeug-0.12.2-4.el8.noarch.r 16 MB/s | 457 kB 00:00 2021-09-16T19:30:19.544 INFO:teuthology.orchestra.run.smithi109.stdout:(28/32): python3-markupsafe-0.23-19.el8.x86_64. 1.8 MB/s | 39 kB 00:00 2021-09-16T19:30:19.563 INFO:teuthology.orchestra.run.smithi109.stdout:(29/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 8.9 MB/s | 238 kB 00:00 2021-09-16T19:30:19.571 INFO:teuthology.orchestra.run.smithi109.stdout:(30/32): python3-pycparser-2.14-14.el8.noarch.r 4.1 MB/s | 109 kB 00:00 2021-09-16T19:30:19.587 INFO:teuthology.orchestra.run.smithi109.stdout:(31/32): python3-cryptography-3.2.1-4.el8.x86_6 23 MB/s | 559 kB 00:00 2021-09-16T19:30:19.736 INFO:teuthology.orchestra.run.smithi109.stdout:(32/32): ceph-osd-17.0.0-7730.gbc749ef4.el8.x86 21 MB/s | 17 MB 00:00 2021-09-16T19:30:19.739 INFO:teuthology.orchestra.run.smithi109.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:30:19.740 INFO:teuthology.orchestra.run.smithi109.stdout:Total 25 MB/s | 28 MB 00:01 2021-09-16T19:30:19.747 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction check 2021-09-16T19:30:19.783 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction check succeeded. 2021-09-16T19:30:19.784 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction test 2021-09-16T19:30:19.985 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction test succeeded. 2021-09-16T19:30:19.988 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction 2021-09-16T19:30:20.834 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:30:20.837 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================================= 2021-09-16T19:30:20.838 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2021-09-16T19:30:20.838 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================================= 2021-09-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2021-09-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: ceph x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 6.4 k 2021-09-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2021-09-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 2.3 M 2021-09-16T19:30:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 1.5 M 2021-09-16T19:30:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core noarch 2:17.0.0-7730.gbc749ef4.el8 ceph-noarch 214 k 2021-09-16T19:30:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 3.9 M 2021-09-16T19:30:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 17 M 2021-09-16T19:30:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-09-16T19:30:20.841 INFO:teuthology.orchestra.run.smithi118.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-09-16T19:30:20.841 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2021-09-16T19:30:20.841 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-09-16T19:30:20.841 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-09-16T19:30:20.841 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography x86_64 3.2.1-4.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2021-09-16T19:30:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-09-16T19:30:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-09-16T19:30:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-09-16T19:30:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2021-09-16T19:30:20.843 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2021-09-16T19:30:20.843 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-09-16T19:30:20.843 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-09-16T19:30:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-09-16T19:30:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2021-09-16T19:30:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2021-09-16T19:30:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2021-09-16T19:30:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-09-16T19:30:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-09-16T19:30:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-09-16T19:30:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-09-16T19:30:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-09-16T19:30:20.846 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-09-16T19:30:20.846 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-09-16T19:30:20.846 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2021-09-16T19:30:20.846 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-09-16T19:30:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:30:20.847 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2021-09-16T19:30:20.847 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================================= 2021-09-16T19:30:20.847 INFO:teuthology.orchestra.run.smithi118.stdout:Install 32 Packages 2021-09-16T19:30:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:30:20.849 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 28 M 2021-09-16T19:30:20.849 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 112 M 2021-09-16T19:30:20.849 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2021-09-16T19:30:20.943 INFO:teuthology.orchestra.run.smithi118.stdout:(1/32): ceph-17.0.0-7730.gbc749ef4.el8.x86_64.r 65 kB/s | 6.4 kB 00:00 2021-09-16T19:30:21.200 INFO:teuthology.orchestra.run.smithi118.stdout:(2/32): ceph-mgr-17.0.0-7730.gbc749ef4.el8.x86_ 4.3 MB/s | 1.5 MB 00:00 2021-09-16T19:30:21.211 INFO:teuthology.orchestra.run.smithi118.stdout:(3/32): ceph-mds-17.0.0-7730.gbc749ef4.el8.x86_ 6.1 MB/s | 2.3 MB 00:00 2021-09-16T19:30:21.341 INFO:teuthology.orchestra.run.smithi118.stdout:(4/32): ceph-mgr-modules-core-17.0.0-7730.gbc74 1.6 MB/s | 214 kB 00:00 2021-09-16T19:30:21.393 INFO:teuthology.orchestra.run.smithi118.stdout:(5/32): ceph-mon-17.0.0-7730.gbc749ef4.el8.x86_ 8.6 MB/s | 3.9 MB 00:00 2021-09-16T19:30:21.509 INFO:teuthology.orchestra.run.smithi118.stdout:(6/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 260 kB/s | 44 kB 00:00 2021-09-16T19:30:21.532 INFO:teuthology.orchestra.run.smithi118.stdout:(7/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 1.3 MB/s | 185 kB 00:00 2021-09-16T19:30:21.609 INFO:teuthology.orchestra.run.smithi118.stdout:(8/32): python3-cherrypy-18.4.0-1.el8.noarch.rp 4.9 MB/s | 384 kB 00:00 2021-09-16T19:30:21.692 INFO:teuthology.orchestra.run.smithi118.stdout:(9/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 945 kB/s | 173 kB 00:00 2021-09-16T19:30:21.693 INFO:teuthology.orchestra.run.smithi118.stdout:(10/32): python3-jaraco-6.2-6.el8.noarch.rpm 134 kB/s | 11 kB 00:00 2021-09-16T19:30:21.756 INFO:teuthology.orchestra.run.smithi118.stdout:(11/32): python3-jaraco-functools-2.0-4.el8.noa 275 kB/s | 18 kB 00:00 2021-09-16T19:30:21.814 INFO:teuthology.orchestra.run.smithi118.stdout:(12/32): python3-logutils-0.3.5-11.el8.noarch.r 400 kB/s | 49 kB 00:00 2021-09-16T19:30:21.869 INFO:teuthology.orchestra.run.smithi118.stdout:(13/32): python3-more-itertools-7.2.0-3.el8.noa 522 kB/s | 59 kB 00:00 2021-09-16T19:30:21.945 INFO:teuthology.orchestra.run.smithi118.stdout:(14/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.1 MB/s | 283 kB 00:00 2021-09-16T19:30:21.945 INFO:teuthology.orchestra.run.smithi118.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 204 kB/s | 16 kB 00:00 2021-09-16T19:30:22.015 INFO:teuthology.orchestra.run.smithi118.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 276 kB/s | 19 kB 00:00 2021-09-16T19:30:22.015 INFO:teuthology.orchestra.run.smithi118.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 352 kB/s | 24 kB 00:00 2021-09-16T19:30:22.236 INFO:teuthology.orchestra.run.smithi118.stdout:(18/32): ceph-osd-17.0.0-7730.gbc749ef4.el8.x86 16 MB/s | 17 MB 00:01 2021-09-16T19:30:22.313 INFO:teuthology.orchestra.run.smithi118.stdout:(19/32): python3-tempora-1.14.1-5.el8.noarch.rp 98 kB/s | 29 kB 00:00 2021-09-16T19:30:22.313 INFO:teuthology.orchestra.run.smithi118.stdout:(20/32): python3-trustme-0.6.0-4.el8.noarch.rpm 92 kB/s | 27 kB 00:00 2021-09-16T19:30:22.339 INFO:teuthology.orchestra.run.smithi118.stdout:(21/32): python3-webtest-2.0.33-1.el8.noarch.rp 3.3 MB/s | 85 kB 00:00 2021-09-16T19:30:22.364 INFO:teuthology.orchestra.run.smithi118.stdout:(22/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.8 MB/s | 251 kB 00:00 2021-09-16T19:30:22.365 INFO:teuthology.orchestra.run.smithi118.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 896 kB/s | 23 kB 00:00 2021-09-16T19:30:22.404 INFO:teuthology.orchestra.run.smithi118.stdout:(24/32): python3-mako-1.0.6-13.el8.noarch.rpm 3.9 MB/s | 157 kB 00:00 2021-09-16T19:30:22.411 INFO:teuthology.orchestra.run.smithi118.stdout:(25/32): python3-pytz-2017.2-9.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2021-09-16T19:30:22.429 INFO:teuthology.orchestra.run.smithi118.stdout:(26/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 4.2 MB/s | 103 kB 00:00 2021-09-16T19:30:22.439 INFO:teuthology.orchestra.run.smithi118.stdout:(27/32): python3-werkzeug-0.12.2-4.el8.noarch.r 16 MB/s | 457 kB 00:00 2021-09-16T19:30:22.451 INFO:teuthology.orchestra.run.smithi118.stdout:(28/32): python3-waitress-1.2.1-2.el8.1.noarch. 1.1 MB/s | 241 kB 00:00 2021-09-16T19:30:22.451 INFO:teuthology.orchestra.run.smithi118.stdout:(29/32): python3-markupsafe-0.23-19.el8.x86_64. 1.7 MB/s | 39 kB 00:00 2021-09-16T19:30:22.468 INFO:teuthology.orchestra.run.smithi118.stdout:(30/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 9.4 MB/s | 238 kB 00:00 2021-09-16T19:30:22.484 INFO:teuthology.orchestra.run.smithi118.stdout:(31/32): python3-pycparser-2.14-14.el8.noarch.r 3.2 MB/s | 109 kB 00:00 2021-09-16T19:30:22.492 INFO:teuthology.orchestra.run.smithi118.stdout:(32/32): python3-cryptography-3.2.1-4.el8.x86_6 13 MB/s | 559 kB 00:00 2021-09-16T19:30:22.494 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:30:22.494 INFO:teuthology.orchestra.run.smithi118.stdout:Total 17 MB/s | 28 MB 00:01 2021-09-16T19:30:22.501 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2021-09-16T19:30:22.537 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2021-09-16T19:30:22.538 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2021-09-16T19:30:22.737 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2021-09-16T19:30:22.741 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2021-09-16T19:30:23.514 INFO:teuthology.orchestra.run.smithi109.stdout: Preparing : 1/1 2021-09-16T19:30:23.796 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-09-16T19:30:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2021-09-16T19:30:24.051 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-09-16T19:30:24.252 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-09-16T19:30:24.308 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 3/32 2021-09-16T19:30:24.490 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-09-16T19:30:24.664 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 4/32 2021-09-16T19:30:24.739 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 3/32 2021-09-16T19:30:24.958 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 5/32 2021-09-16T19:30:25.120 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 4/32 2021-09-16T19:30:25.154 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 6/32 2021-09-16T19:30:25.338 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 7/32 2021-09-16T19:30:25.430 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 5/32 2021-09-16T19:30:25.575 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 8/32 2021-09-16T19:30:25.635 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 6/32 2021-09-16T19:30:25.795 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 7/32 2021-09-16T19:30:25.796 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 9/32 2021-09-16T19:30:26.050 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 8/32 2021-09-16T19:30:26.108 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 10/32 2021-09-16T19:30:26.293 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2021-09-16T19:30:26.300 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 9/32 2021-09-16T19:30:26.486 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 12/32 2021-09-16T19:30:26.616 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 10/32 2021-09-16T19:30:26.744 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2021-09-16T19:30:26.824 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2021-09-16T19:30:26.968 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 14/32 2021-09-16T19:30:27.035 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 12/32 2021-09-16T19:30:27.182 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2021-09-16T19:30:27.310 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2021-09-16T19:30:27.347 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2021-09-16T19:30:27.547 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2021-09-16T19:30:27.558 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 14/32 2021-09-16T19:30:27.726 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2021-09-16T19:30:27.764 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2021-09-16T19:30:27.956 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2021-09-16T19:30:28.158 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2021-09-16T19:30:28.385 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-09-16T19:30:28.565 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2021-09-16T19:30:28.668 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-09-16T19:30:28.762 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2021-09-16T19:30:28.907 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2021-09-16T19:30:28.943 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-09-16T19:30:29.170 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2021-09-16T19:30:29.176 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2021-09-16T19:30:29.406 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2021-09-16T19:30:29.509 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2021-09-16T19:30:29.615 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-09-16T19:30:29.893 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-09-16T19:30:29.899 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-09-16T19:30:30.167 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-09-16T19:30:30.264 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-7730.gbc749ef4.el8. 27/32 2021-09-16T19:30:30.415 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2021-09-16T19:30:30.428 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-mgr-2:17.0.0-7730.gbc749ef4.el8.x86_64 28/32 2021-09-16T19:30:30.752 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2021-09-16T19:30:31.134 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-09-16T19:30:31.439 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-7730.gbc749ef4.el8. 27/32 2021-09-16T19:30:31.621 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-2:17.0.0-7730.gbc749ef4.el8.x86_64 28/32 2021-09-16T19:30:31.909 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: ceph-mgr-2:17.0.0-7730.gbc749ef4.el8.x86_64 28/32 2021-09-16T19:30:32.069 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-osd-2:17.0.0-7730.gbc749ef4.el8.x86_64 29/32 2021-09-16T19:30:32.458 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: ceph-osd-2:17.0.0-7730.gbc749ef4.el8.x86_64 29/32 2021-09-16T19:30:32.652 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-mon-2:17.0.0-7730.gbc749ef4.el8.x86_64 30/32 2021-09-16T19:30:32.870 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: ceph-mon-2:17.0.0-7730.gbc749ef4.el8.x86_64 30/32 2021-09-16T19:30:33.019 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-mds-2:17.0.0-7730.gbc749ef4.el8.x86_64 31/32 2021-09-16T19:30:33.037 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: ceph-mds-2:17.0.0-7730.gbc749ef4.el8.x86_64 31/32 2021-09-16T19:30:33.105 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-2:17.0.0-7730.gbc749ef4.el8.x86_64 28/32 2021-09-16T19:30:33.142 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-2:17.0.0-7730.gbc749ef4.el8.x86_64 32/32 2021-09-16T19:30:33.279 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-osd-2:17.0.0-7730.gbc749ef4.el8.x86_64 29/32 2021-09-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: ceph-2:17.0.0-7730.gbc749ef4.el8.x86_64 32/32 2021-09-16T19:30:33.519 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/32 2021-09-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-mds-2:17.0.0-7730.gbc749ef4.el8.x86_64 2/32 2021-09-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-mgr-2:17.0.0-7730.gbc749ef4.el8.x86_64 3/32 2021-09-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-mon-2:17.0.0-7730.gbc749ef4.el8.x86_64 4/32 2021-09-16T19:30:33.520 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-osd-2:17.0.0-7730.gbc749ef4.el8.x86_64 5/32 2021-09-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-7730.gbc749ef4.el8. 6/32 2021-09-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2021-09-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2021-09-16T19:30:33.521 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2021-09-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2021-09-16T19:30:33.522 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2021-09-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2021-09-16T19:30:33.523 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2021-09-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2021-09-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2021-09-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2021-09-16T19:30:33.525 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2021-09-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2021-09-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2021-09-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2021-09-16T19:30:33.526 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 21/32 2021-09-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2021-09-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2021-09-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2021-09-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 25/32 2021-09-16T19:30:33.527 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 26/32 2021-09-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 27/32 2021-09-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/32 2021-09-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2021-09-16T19:30:33.528 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 30/32 2021-09-16T19:30:33.529 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 31/32 2021-09-16T19:30:33.669 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-osd-2:17.0.0-7730.gbc749ef4.el8.x86_64 29/32 2021-09-16T19:30:33.866 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mon-2:17.0.0-7730.gbc749ef4.el8.x86_64 30/32 2021-09-16T19:30:34.083 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mon-2:17.0.0-7730.gbc749ef4.el8.x86_64 30/32 2021-09-16T19:30:34.234 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 32/32 2021-09-16T19:30:34.234 INFO:teuthology.orchestra.run.smithi109.stdout:Installed products updated. 2021-09-16T19:30:34.257 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mds-2:17.0.0-7730.gbc749ef4.el8.x86_64 31/32 2021-09-16T19:30:34.276 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mds-2:17.0.0-7730.gbc749ef4.el8.x86_64 31/32 2021-09-16T19:30:34.372 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-2:17.0.0-7730.gbc749ef4.el8.x86_64 32/32 2021-09-16T19:30:34.719 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-2:17.0.0-7730.gbc749ef4.el8.x86_64 32/32 2021-09-16T19:30:34.719 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/32 2021-09-16T19:30:34.719 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:17.0.0-7730.gbc749ef4.el8.x86_64 2/32 2021-09-16T19:30:34.720 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-2:17.0.0-7730.gbc749ef4.el8.x86_64 3/32 2021-09-16T19:30:34.720 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:17.0.0-7730.gbc749ef4.el8.x86_64 4/32 2021-09-16T19:30:34.720 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-osd-2:17.0.0-7730.gbc749ef4.el8.x86_64 5/32 2021-09-16T19:30:34.720 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-7730.gbc749ef4.el8. 6/32 2021-09-16T19:30:34.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2021-09-16T19:30:34.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2021-09-16T19:30:34.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2021-09-16T19:30:34.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2021-09-16T19:30:34.721 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2021-09-16T19:30:34.722 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2021-09-16T19:30:34.722 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2021-09-16T19:30:34.722 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2021-09-16T19:30:34.722 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2021-09-16T19:30:34.723 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2021-09-16T19:30:34.723 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2021-09-16T19:30:34.723 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2021-09-16T19:30:34.723 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2021-09-16T19:30:34.724 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2021-09-16T19:30:34.724 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 21/32 2021-09-16T19:30:34.724 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2021-09-16T19:30:34.724 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2021-09-16T19:30:34.724 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2021-09-16T19:30:34.725 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 25/32 2021-09-16T19:30:34.725 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 26/32 2021-09-16T19:30:34.725 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 27/32 2021-09-16T19:30:34.725 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/32 2021-09-16T19:30:34.726 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2021-09-16T19:30:34.726 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 30/32 2021-09-16T19:30:34.726 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 31/32 2021-09-16T19:30:35.415 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 32/32 2021-09-16T19:30:35.416 INFO:teuthology.orchestra.run.smithi118.stdout:Installed products updated. 2021-09-16T19:30:44.648 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:29:57 PM UTC. 2021-09-16T19:30:45.799 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:29:59 PM UTC. 2021-09-16T19:30:47.108 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:30:47.108 INFO:teuthology.orchestra.run.smithi109.stdout:Installed: 2021-09-16T19:30:47.108 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:30:47.109 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-mds-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:30:47.109 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-mgr-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:30:47.109 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-mgr-modules-core-2:17.0.0-7730.gbc749ef4.el8.noarch 2021-09-16T19:30:47.110 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-mon-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:30:47.110 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-osd-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:30:47.110 INFO:teuthology.orchestra.run.smithi109.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-09-16T19:30:47.111 INFO:teuthology.orchestra.run.smithi109.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-09-16T19:30:47.111 INFO:teuthology.orchestra.run.smithi109.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-09-16T19:30:47.111 INFO:teuthology.orchestra.run.smithi109.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-09-16T19:30:47.111 INFO:teuthology.orchestra.run.smithi109.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-09-16T19:30:47.111 INFO:teuthology.orchestra.run.smithi109.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2021-09-16T19:30:47.112 INFO:teuthology.orchestra.run.smithi109.stdout: python3-jaraco-6.2-6.el8.noarch 2021-09-16T19:30:47.112 INFO:teuthology.orchestra.run.smithi109.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-09-16T19:30:47.112 INFO:teuthology.orchestra.run.smithi109.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-09-16T19:30:47.112 INFO:teuthology.orchestra.run.smithi109.stdout: python3-mako-1.0.6-13.el8.noarch 2021-09-16T19:30:47.113 INFO:teuthology.orchestra.run.smithi109.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-09-16T19:30:47.113 INFO:teuthology.orchestra.run.smithi109.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-09-16T19:30:47.113 INFO:teuthology.orchestra.run.smithi109.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-09-16T19:30:47.114 INFO:teuthology.orchestra.run.smithi109.stdout: python3-portend-2.6-1.el8.noarch 2021-09-16T19:30:47.114 INFO:teuthology.orchestra.run.smithi109.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-09-16T19:30:47.114 INFO:teuthology.orchestra.run.smithi109.stdout: python3-pycparser-2.14-14.el8.noarch 2021-09-16T19:30:47.115 INFO:teuthology.orchestra.run.smithi109.stdout: python3-pytz-2017.2-9.el8.noarch 2021-09-16T19:30:47.115 INFO:teuthology.orchestra.run.smithi109.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-09-16T19:30:47.115 INFO:teuthology.orchestra.run.smithi109.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-09-16T19:30:47.115 INFO:teuthology.orchestra.run.smithi109.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-09-16T19:30:47.116 INFO:teuthology.orchestra.run.smithi109.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-09-16T19:30:47.116 INFO:teuthology.orchestra.run.smithi109.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-09-16T19:30:47.116 INFO:teuthology.orchestra.run.smithi109.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-09-16T19:30:47.117 INFO:teuthology.orchestra.run.smithi109.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-09-16T19:30:47.117 INFO:teuthology.orchestra.run.smithi109.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-09-16T19:30:47.117 INFO:teuthology.orchestra.run.smithi109.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-09-16T19:30:47.117 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:30:47.117 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:30:47.261 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install ceph-base 2021-09-16T19:30:47.594 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:30:48.084 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-09-16T19:30:48.333 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:30:48.334 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2021-09-16T19:30:48.334 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:30:48.334 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:30:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:30:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core-2:17.0.0-7730.gbc749ef4.el8.noarch 2021-09-16T19:30:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:30:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:30:48.335 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-09-16T19:30:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-09-16T19:30:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-09-16T19:30:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-09-16T19:30:48.336 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-09-16T19:30:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2021-09-16T19:30:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-6.2-6.el8.noarch 2021-09-16T19:30:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-09-16T19:30:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-09-16T19:30:48.337 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako-1.0.6-13.el8.noarch 2021-09-16T19:30:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-09-16T19:30:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-09-16T19:30:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-09-16T19:30:48.338 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend-2.6-1.el8.noarch 2021-09-16T19:30:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-09-16T19:30:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser-2.14-14.el8.noarch 2021-09-16T19:30:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz-2017.2-9.el8.noarch 2021-09-16T19:30:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-09-16T19:30:48.339 INFO:teuthology.orchestra.run.smithi118.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-09-16T19:30:48.340 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-09-16T19:30:48.340 INFO:teuthology.orchestra.run.smithi118.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-09-16T19:30:48.340 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-09-16T19:30:48.340 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-09-16T19:30:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-09-16T19:30:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-09-16T19:30:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-09-16T19:30:48.341 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:30:48.341 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:30:48.472 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-base 2021-09-16T19:30:48.800 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:30:49.305 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-09-16T19:30:52.942 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-09-16T19:30:54.181 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 77 kB/s | 2.4 kB 00:00 2021-09-16T19:30:58.263 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:04 ago on Thu 16 Sep 2021 07:28:26 PM UTC. 2021-09-16T19:30:59.509 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:28:27 PM UTC. 2021-09-16T19:31:00.353 INFO:teuthology.orchestra.run.smithi109.stdout:Package ceph-base-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:31:00.789 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:31:00.790 INFO:teuthology.orchestra.run.smithi109.stdout:Nothing to do. 2021-09-16T19:31:00.790 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:31:00.901 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install cephadm 2021-09-16T19:31:01.226 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:31:01.597 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-base-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:31:01.702 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-09-16T19:31:01.872 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 84 kB/s | 2.4 kB 00:00 2021-09-16T19:31:02.018 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:31:02.019 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2021-09-16T19:31:02.019 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:31:02.132 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install cephadm 2021-09-16T19:31:02.456 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:31:02.943 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 100 kB/s | 2.8 kB 00:00 2021-09-16T19:31:03.115 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-09-16T19:31:03.725 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:31:03.727 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:31:03.727 INFO:teuthology.orchestra.run.smithi109.stdout: Package Arch Version Repository Size 2021-09-16T19:31:03.728 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:31:03.728 INFO:teuthology.orchestra.run.smithi109.stdout:Installing: 2021-09-16T19:31:03.729 INFO:teuthology.orchestra.run.smithi109.stdout: cephadm noarch 2:17.0.0-7730.gbc749ef4.el8 ceph-noarch 70 k 2021-09-16T19:31:03.729 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:31:03.729 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction Summary 2021-09-16T19:31:03.730 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:31:03.730 INFO:teuthology.orchestra.run.smithi109.stdout:Install 1 Package 2021-09-16T19:31:03.731 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:31:03.731 INFO:teuthology.orchestra.run.smithi109.stdout:Total download size: 70 k 2021-09-16T19:31:03.731 INFO:teuthology.orchestra.run.smithi109.stdout:Installed size: 310 k 2021-09-16T19:31:03.732 INFO:teuthology.orchestra.run.smithi109.stdout:Downloading Packages: 2021-09-16T19:31:03.878 INFO:teuthology.orchestra.run.smithi109.stdout:cephadm-17.0.0-7730.gbc749ef4.el8.noarch.rpm 459 kB/s | 70 kB 00:00 2021-09-16T19:31:03.879 INFO:teuthology.orchestra.run.smithi109.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:31:03.880 INFO:teuthology.orchestra.run.smithi109.stdout:Total 454 kB/s | 70 kB 00:00 2021-09-16T19:31:03.880 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction check 2021-09-16T19:31:03.887 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction check succeeded. 2021-09-16T19:31:03.888 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction test 2021-09-16T19:31:03.889 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction test succeeded. 2021-09-16T19:31:03.893 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction 2021-09-16T19:31:04.610 INFO:teuthology.orchestra.run.smithi109.stdout: Preparing : 1/1 2021-09-16T19:31:05.661 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:31:05.662 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:31:05.662 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2021-09-16T19:31:05.662 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:31:05.662 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2021-09-16T19:31:05.663 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm noarch 2:17.0.0-7730.gbc749ef4.el8 ceph-noarch 70 k 2021-09-16T19:31:05.663 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:31:05.663 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2021-09-16T19:31:05.663 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:31:05.664 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2021-09-16T19:31:05.664 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:31:05.664 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 70 k 2021-09-16T19:31:05.664 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 310 k 2021-09-16T19:31:05.664 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2021-09-16T19:31:05.975 INFO:teuthology.orchestra.run.smithi118.stdout:cephadm-17.0.0-7730.gbc749ef4.el8.noarch.rpm 223 kB/s | 70 kB 00:00 2021-09-16T19:31:05.976 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:31:05.976 INFO:teuthology.orchestra.run.smithi118.stdout:Total 222 kB/s | 70 kB 00:00 2021-09-16T19:31:05.978 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2021-09-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2021-09-16T19:31:05.983 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2021-09-16T19:31:05.984 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2021-09-16T19:31:05.988 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2021-09-16T19:31:06.037 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2021-09-16T19:31:07.004 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 1/1 2021-09-16T19:31:07.302 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 1/1 2021-09-16T19:31:07.463 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 1/1 2021-09-16T19:31:07.691 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 1/1 2021-09-16T19:31:07.691 INFO:teuthology.orchestra.run.smithi109.stdout:Installed products updated. 2021-09-16T19:31:08.343 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 1/1 2021-09-16T19:31:08.594 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 1/1 2021-09-16T19:31:08.757 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 1/1 2021-09-16T19:31:08.985 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 1/1 2021-09-16T19:31:08.985 INFO:teuthology.orchestra.run.smithi118.stdout:Installed products updated. 2021-09-16T19:31:18.026 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:04 ago on Thu 16 Sep 2021 07:30:39 PM UTC. 2021-09-16T19:31:19.240 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:30:40 PM UTC. 2021-09-16T19:31:20.511 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:31:20.511 INFO:teuthology.orchestra.run.smithi109.stdout:Installed: 2021-09-16T19:31:20.511 INFO:teuthology.orchestra.run.smithi109.stdout: cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 2021-09-16T19:31:20.512 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:31:20.512 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:31:20.644 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install ceph-immutable-object-cache 2021-09-16T19:31:20.980 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:31:21.483 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-09-16T19:31:21.746 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:31:21.746 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2021-09-16T19:31:21.747 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 2021-09-16T19:31:21.747 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:31:21.747 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:31:21.873 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-immutable-object-cache 2021-09-16T19:31:22.209 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:31:22.697 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 90 kB/s | 2.8 kB 00:00 2021-09-16T19:31:26.358 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.4 kB 00:00 2021-09-16T19:31:27.595 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-09-16T19:31:31.678 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:28:26 PM UTC. 2021-09-16T19:31:32.839 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:28:27 PM UTC. 2021-09-16T19:31:34.158 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:31:34.160 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:31:34.160 INFO:teuthology.orchestra.run.smithi109.stdout: Package Arch Version Repo Size 2021-09-16T19:31:34.161 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:31:34.161 INFO:teuthology.orchestra.run.smithi109.stdout:Installing: 2021-09-16T19:31:34.161 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 169 k 2021-09-16T19:31:34.162 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:31:34.162 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction Summary 2021-09-16T19:31:34.163 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:31:34.163 INFO:teuthology.orchestra.run.smithi109.stdout:Install 1 Package 2021-09-16T19:31:34.164 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:31:34.164 INFO:teuthology.orchestra.run.smithi109.stdout:Total download size: 169 k 2021-09-16T19:31:34.164 INFO:teuthology.orchestra.run.smithi109.stdout:Installed size: 567 k 2021-09-16T19:31:34.165 INFO:teuthology.orchestra.run.smithi109.stdout:Downloading Packages: 2021-09-16T19:31:34.335 INFO:teuthology.orchestra.run.smithi109.stdout:ceph-immutable-object-cache-17.0.0-7730.gbc749e 965 kB/s | 169 kB 00:00 2021-09-16T19:31:34.336 INFO:teuthology.orchestra.run.smithi109.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:31:34.336 INFO:teuthology.orchestra.run.smithi109.stdout:Total 956 kB/s | 169 kB 00:00 2021-09-16T19:31:34.337 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction check 2021-09-16T19:31:34.348 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction check succeeded. 2021-09-16T19:31:34.349 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction test 2021-09-16T19:31:34.400 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction test succeeded. 2021-09-16T19:31:34.404 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction 2021-09-16T19:31:34.531 INFO:teuthology.orchestra.run.smithi109.stdout: Preparing : 1/1 2021-09-16T19:31:34.728 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-7730.gbc749ef4. 1/1 2021-09-16T19:31:35.057 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-7730.gbc749ef4. 1/1 2021-09-16T19:31:35.270 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-7730.gbc749ef4. 1/1 2021-09-16T19:31:35.271 INFO:teuthology.orchestra.run.smithi109.stdout:Installed products updated. 2021-09-16T19:31:35.371 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2021-09-16T19:31:35.372 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2021-09-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 169 k 2021-09-16T19:31:35.373 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:31:35.374 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2021-09-16T19:31:35.374 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2021-09-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 169 k 2021-09-16T19:31:35.375 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 567 k 2021-09-16T19:31:35.376 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2021-09-16T19:31:35.546 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-immutable-object-cache-17.0.0-7730.gbc749e 973 kB/s | 169 kB 00:00 2021-09-16T19:31:35.547 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:31:35.547 INFO:teuthology.orchestra.run.smithi118.stdout:Total 966 kB/s | 169 kB 00:00 2021-09-16T19:31:35.547 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2021-09-16T19:31:35.559 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2021-09-16T19:31:35.559 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2021-09-16T19:31:35.610 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2021-09-16T19:31:35.615 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2021-09-16T19:31:35.753 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2021-09-16T19:31:35.964 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-7730.gbc749ef4. 1/1 2021-09-16T19:31:36.337 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-7730.gbc749ef4. 1/1 2021-09-16T19:31:36.561 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-7730.gbc749ef4. 1/1 2021-09-16T19:31:36.562 INFO:teuthology.orchestra.run.smithi118.stdout:Installed products updated. 2021-09-16T19:31:45.637 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:31:12 PM UTC. 2021-09-16T19:31:47.030 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:31:14 PM UTC. 2021-09-16T19:31:48.148 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:31:48.148 INFO:teuthology.orchestra.run.smithi109.stdout:Installed: 2021-09-16T19:31:48.148 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-immutable-object-cache-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:31:48.149 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:31:48.149 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:31:48.286 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install ceph-mgr 2021-09-16T19:31:48.615 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:31:49.139 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-09-16T19:31:49.517 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:31:49.519 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2021-09-16T19:31:49.519 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:31:49.520 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:31:49.520 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:31:49.651 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr 2021-09-16T19:31:49.978 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:31:50.487 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 90 kB/s | 2.8 kB 00:00 2021-09-16T19:31:53.990 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-09-16T19:31:55.461 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.4 kB 00:00 2021-09-16T19:31:59.287 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:04 ago on Thu 16 Sep 2021 07:28:26 PM UTC. 2021-09-16T19:32:00.783 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:28:27 PM UTC. 2021-09-16T19:32:01.387 INFO:teuthology.orchestra.run.smithi109.stdout:Package ceph-mgr-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:32:01.810 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:32:01.811 INFO:teuthology.orchestra.run.smithi109.stdout:Nothing to do. 2021-09-16T19:32:01.811 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:32:01.917 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install ceph-mgr-dashboard 2021-09-16T19:32:02.243 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:32:02.730 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-09-16T19:32:02.887 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-mgr-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:32:02.906 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-09-16T19:32:03.317 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:32:03.318 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2021-09-16T19:32:03.318 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:32:03.438 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-dashboard 2021-09-16T19:32:03.763 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:32:04.247 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-09-16T19:32:04.422 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-09-16T19:32:04.784 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:32:04.784 INFO:teuthology.orchestra.run.smithi109.stdout:=================================================================================================== 2021-09-16T19:32:04.785 INFO:teuthology.orchestra.run.smithi109.stdout: Package Arch Version Repository Size 2021-09-16T19:32:04.785 INFO:teuthology.orchestra.run.smithi109.stdout:=================================================================================================== 2021-09-16T19:32:04.785 INFO:teuthology.orchestra.run.smithi109.stdout:Installing: 2021-09-16T19:32:04.785 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-mgr-dashboard noarch 2:17.0.0-7730.gbc749ef4.el8 ceph-noarch 1.5 M 2021-09-16T19:32:04.786 INFO:teuthology.orchestra.run.smithi109.stdout:Installing dependencies: 2021-09-16T19:32:04.786 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-grafana-dashboards noarch 2:17.0.0-7730.gbc749ef4.el8 ceph-noarch 24 k 2021-09-16T19:32:04.786 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-prometheus-alerts noarch 2:17.0.0-7730.gbc749ef4.el8 ceph-noarch 8.9 k 2021-09-16T19:32:04.786 INFO:teuthology.orchestra.run.smithi109.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-09-16T19:32:04.787 INFO:teuthology.orchestra.run.smithi109.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-09-16T19:32:04.787 INFO:teuthology.orchestra.run.smithi109.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-09-16T19:32:04.787 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:32:04.787 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction Summary 2021-09-16T19:32:04.787 INFO:teuthology.orchestra.run.smithi109.stdout:=================================================================================================== 2021-09-16T19:32:04.788 INFO:teuthology.orchestra.run.smithi109.stdout:Install 6 Packages 2021-09-16T19:32:04.788 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:32:04.789 INFO:teuthology.orchestra.run.smithi109.stdout:Total download size: 1.8 M 2021-09-16T19:32:04.789 INFO:teuthology.orchestra.run.smithi109.stdout:Installed size: 7.4 M 2021-09-16T19:32:04.789 INFO:teuthology.orchestra.run.smithi109.stdout:Downloading Packages: 2021-09-16T19:32:04.879 INFO:teuthology.orchestra.run.smithi109.stdout:(1/6): ceph-prometheus-alerts-17.0.0-7730.gbc74 98 kB/s | 8.9 kB 00:00 2021-09-16T19:32:04.920 INFO:teuthology.orchestra.run.smithi109.stdout:(2/6): ceph-grafana-dashboards-17.0.0-7730.gbc7 180 kB/s | 24 kB 00:00 2021-09-16T19:32:05.091 INFO:teuthology.orchestra.run.smithi109.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 158 kB/s | 34 kB 00:00 2021-09-16T19:32:05.121 INFO:teuthology.orchestra.run.smithi109.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2021-09-16T19:32:05.144 INFO:teuthology.orchestra.run.smithi109.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 879 kB/s | 196 kB 00:00 2021-09-16T19:32:05.300 INFO:teuthology.orchestra.run.smithi109.stdout:(6/6): ceph-mgr-dashboard-17.0.0-7730.gbc749ef4 3.0 MB/s | 1.5 MB 00:00 2021-09-16T19:32:05.302 INFO:teuthology.orchestra.run.smithi109.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:32:05.303 INFO:teuthology.orchestra.run.smithi109.stdout:Total 3.5 MB/s | 1.8 MB 00:00 2021-09-16T19:32:05.304 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction check 2021-09-16T19:32:05.312 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction check succeeded. 2021-09-16T19:32:05.313 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction test 2021-09-16T19:32:05.365 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction test succeeded. 2021-09-16T19:32:05.369 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction 2021-09-16T19:32:05.641 INFO:teuthology.orchestra.run.smithi109.stdout: Preparing : 1/1 2021-09-16T19:32:05.837 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2021-09-16T19:32:06.057 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2021-09-16T19:32:06.212 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:32:06.213 INFO:teuthology.orchestra.run.smithi118.stdout:=================================================================================================== 2021-09-16T19:32:06.213 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2021-09-16T19:32:06.214 INFO:teuthology.orchestra.run.smithi118.stdout:=================================================================================================== 2021-09-16T19:32:06.214 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2021-09-16T19:32:06.214 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard noarch 2:17.0.0-7730.gbc749ef4.el8 ceph-noarch 1.5 M 2021-09-16T19:32:06.215 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2021-09-16T19:32:06.215 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards noarch 2:17.0.0-7730.gbc749ef4.el8 ceph-noarch 24 k 2021-09-16T19:32:06.215 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts noarch 2:17.0.0-7730.gbc749ef4.el8 ceph-noarch 8.9 k 2021-09-16T19:32:06.215 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-09-16T19:32:06.216 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-09-16T19:32:06.216 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-09-16T19:32:06.216 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:32:06.216 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2021-09-16T19:32:06.217 INFO:teuthology.orchestra.run.smithi118.stdout:=================================================================================================== 2021-09-16T19:32:06.217 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2021-09-16T19:32:06.217 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:32:06.218 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 1.8 M 2021-09-16T19:32:06.218 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 7.4 M 2021-09-16T19:32:06.218 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2021-09-16T19:32:06.259 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2021-09-16T19:32:06.315 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): ceph-prometheus-alerts-17.0.0-7730.gbc74 89 kB/s | 8.9 kB 00:00 2021-09-16T19:32:06.326 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): ceph-grafana-dashboards-17.0.0-7730.gbc7 216 kB/s | 24 kB 00:00 2021-09-16T19:32:06.400 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 399 kB/s | 34 kB 00:00 2021-09-16T19:32:06.405 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-7730.gbc749ef4.el8.n 4/6 2021-09-16T19:32:06.429 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2021-09-16T19:32:06.473 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.3 MB/s | 196 kB 00:00 2021-09-16T19:32:06.638 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): ceph-mgr-dashboard-17.0.0-7730.gbc749ef4 3.6 MB/s | 1.5 MB 00:00 2021-09-16T19:32:06.640 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:32:06.641 INFO:teuthology.orchestra.run.smithi118.stdout:Total 4.3 MB/s | 1.8 MB 00:00 2021-09-16T19:32:06.642 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2021-09-16T19:32:06.650 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2021-09-16T19:32:06.651 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2021-09-16T19:32:06.702 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2021-09-16T19:32:06.706 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2021-09-16T19:32:06.752 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-7730.gbc749ef4.el8. 5/6 2021-09-16T19:32:06.960 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2021-09-16T19:32:07.015 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-7730.gbc749ef4.el8.noarc 6/6 2021-09-16T19:32:07.234 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-7730.gbc749ef4.el8.noarc 6/6 2021-09-16T19:32:07.235 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-7730.gbc749ef4.el8. 1/6 2021-09-16T19:32:07.235 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-7730.gbc749ef4.el8.noarc 2/6 2021-09-16T19:32:07.235 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-7730.gbc749ef4.el8.n 3/6 2021-09-16T19:32:07.235 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2021-09-16T19:32:07.236 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2021-09-16T19:32:07.256 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2021-09-16T19:32:07.443 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2021-09-16T19:32:07.497 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2021-09-16T19:32:07.498 INFO:teuthology.orchestra.run.smithi109.stdout:Installed products updated. 2021-09-16T19:32:07.620 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2021-09-16T19:32:07.775 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-7730.gbc749ef4.el8.n 4/6 2021-09-16T19:32:08.094 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-7730.gbc749ef4.el8. 5/6 2021-09-16T19:32:08.322 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-7730.gbc749ef4.el8.noarc 6/6 2021-09-16T19:32:08.527 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-7730.gbc749ef4.el8.noarc 6/6 2021-09-16T19:32:08.527 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-7730.gbc749ef4.el8. 1/6 2021-09-16T19:32:08.527 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-7730.gbc749ef4.el8.noarc 2/6 2021-09-16T19:32:08.528 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-7730.gbc749ef4.el8.n 3/6 2021-09-16T19:32:08.528 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2021-09-16T19:32:08.528 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2021-09-16T19:32:08.783 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2021-09-16T19:32:08.784 INFO:teuthology.orchestra.run.smithi118.stdout:Installed products updated. 2021-09-16T19:32:17.793 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:31:40 PM UTC. 2021-09-16T19:32:19.132 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:04 ago on Thu 16 Sep 2021 07:31:41 PM UTC. 2021-09-16T19:32:20.337 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:32:20.338 INFO:teuthology.orchestra.run.smithi109.stdout:Installed: 2021-09-16T19:32:20.338 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-grafana-dashboards-2:17.0.0-7730.gbc749ef4.el8.noarch 2021-09-16T19:32:20.338 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-mgr-dashboard-2:17.0.0-7730.gbc749ef4.el8.noarch 2021-09-16T19:32:20.339 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-prometheus-alerts-2:17.0.0-7730.gbc749ef4.el8.noarch 2021-09-16T19:32:20.339 INFO:teuthology.orchestra.run.smithi109.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-09-16T19:32:20.339 INFO:teuthology.orchestra.run.smithi109.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-09-16T19:32:20.339 INFO:teuthology.orchestra.run.smithi109.stdout: python3-routes-2.4.1-12.el8.noarch 2021-09-16T19:32:20.339 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:32:20.340 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:32:20.504 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install ceph-mgr-diskprediction-local 2021-09-16T19:32:20.832 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:32:21.331 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 90 kB/s | 2.8 kB 00:00 2021-09-16T19:32:21.637 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:32:21.638 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2021-09-16T19:32:21.638 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards-2:17.0.0-7730.gbc749ef4.el8.noarch 2021-09-16T19:32:21.638 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard-2:17.0.0-7730.gbc749ef4.el8.noarch 2021-09-16T19:32:21.639 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts-2:17.0.0-7730.gbc749ef4.el8.noarch 2021-09-16T19:32:21.639 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-09-16T19:32:21.639 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-09-16T19:32:21.639 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes-2.4.1-12.el8.noarch 2021-09-16T19:32:21.639 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:32:21.640 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:32:21.764 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-diskprediction-local 2021-09-16T19:32:22.093 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:32:22.583 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 93 kB/s | 2.8 kB 00:00 2021-09-16T19:32:26.421 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 75 kB/s | 2.4 kB 00:00 2021-09-16T19:32:27.560 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 74 kB/s | 2.4 kB 00:00 2021-09-16T19:32:31.820 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:28:26 PM UTC. 2021-09-16T19:32:32.810 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:28:27 PM UTC. 2021-09-16T19:32:34.430 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:32:34.432 INFO:teuthology.orchestra.run.smithi109.stdout:====================================================================================================================== 2021-09-16T19:32:34.432 INFO:teuthology.orchestra.run.smithi109.stdout: Package Arch Version Repository Size 2021-09-16T19:32:34.432 INFO:teuthology.orchestra.run.smithi109.stdout:====================================================================================================================== 2021-09-16T19:32:34.432 INFO:teuthology.orchestra.run.smithi109.stdout:Installing: 2021-09-16T19:32:34.433 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-7730.gbc749ef4.el8 ceph-noarch 7.4 M 2021-09-16T19:32:34.433 INFO:teuthology.orchestra.run.smithi109.stdout:Installing dependencies: 2021-09-16T19:32:34.433 INFO:teuthology.orchestra.run.smithi109.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2021-09-16T19:32:34.433 INFO:teuthology.orchestra.run.smithi109.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2021-09-16T19:32:34.433 INFO:teuthology.orchestra.run.smithi109.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2021-09-16T19:32:34.434 INFO:teuthology.orchestra.run.smithi109.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2021-09-16T19:32:34.434 INFO:teuthology.orchestra.run.smithi109.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2021-09-16T19:32:34.434 INFO:teuthology.orchestra.run.smithi109.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2021-09-16T19:32:34.435 INFO:teuthology.orchestra.run.smithi109.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2021-09-16T19:32:34.435 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:32:34.435 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction Summary 2021-09-16T19:32:34.435 INFO:teuthology.orchestra.run.smithi109.stdout:====================================================================================================================== 2021-09-16T19:32:34.435 INFO:teuthology.orchestra.run.smithi109.stdout:Install 8 Packages 2021-09-16T19:32:34.436 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:32:34.436 INFO:teuthology.orchestra.run.smithi109.stdout:Total download size: 36 M 2021-09-16T19:32:34.437 INFO:teuthology.orchestra.run.smithi109.stdout:Installed size: 199 M 2021-09-16T19:32:34.437 INFO:teuthology.orchestra.run.smithi109.stdout:Downloading Packages: 2021-09-16T19:32:34.508 INFO:teuthology.orchestra.run.smithi109.stdout:(1/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 50 MB/s | 3.7 MB 00:00 2021-09-16T19:32:34.567 INFO:teuthology.orchestra.run.smithi109.stdout:(2/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 81 MB/s | 4.7 MB 00:00 2021-09-16T19:32:34.704 INFO:teuthology.orchestra.run.smithi109.stdout:(3/8): python3-scipy-1.0.0-20.module+el8.1.0+33 54 MB/s | 15 MB 00:00 2021-09-16T19:32:34.727 INFO:teuthology.orchestra.run.smithi109.stdout:(4/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 9.7 MB/s | 225 kB 00:00 2021-09-16T19:32:34.748 INFO:teuthology.orchestra.run.smithi109.stdout:(5/8): openblas-0.3.12-1.el8.x86_64.rpm 25 MB/s | 4.6 MB 00:00 2021-09-16T19:32:34.761 INFO:teuthology.orchestra.run.smithi109.stdout:(6/8): libquadmath-8.4.1-1.el8.x86_64.rpm 4.9 MB/s | 169 kB 00:00 2021-09-16T19:32:34.778 INFO:teuthology.orchestra.run.smithi109.stdout:(7/8): libgfortran-8.4.1-1.el8.x86_64.rpm 25 MB/s | 639 kB 00:00 2021-09-16T19:32:34.989 INFO:teuthology.orchestra.run.smithi109.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-773 13 MB/s | 7.4 MB 00:00 2021-09-16T19:32:34.991 INFO:teuthology.orchestra.run.smithi109.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:32:34.991 INFO:teuthology.orchestra.run.smithi109.stdout:Total 64 MB/s | 36 MB 00:00 2021-09-16T19:32:35.001 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction check 2021-09-16T19:32:35.020 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction check succeeded. 2021-09-16T19:32:35.020 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction test 2021-09-16T19:32:35.315 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction test succeeded. 2021-09-16T19:32:35.319 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction 2021-09-16T19:32:35.374 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:32:35.375 INFO:teuthology.orchestra.run.smithi118.stdout:====================================================================================================================== 2021-09-16T19:32:35.375 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2021-09-16T19:32:35.376 INFO:teuthology.orchestra.run.smithi118.stdout:====================================================================================================================== 2021-09-16T19:32:35.376 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2021-09-16T19:32:35.376 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-7730.gbc749ef4.el8 ceph-noarch 7.4 M 2021-09-16T19:32:35.376 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2021-09-16T19:32:35.377 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2021-09-16T19:32:35.377 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2021-09-16T19:32:35.377 INFO:teuthology.orchestra.run.smithi118.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2021-09-16T19:32:35.377 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2021-09-16T19:32:35.377 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2021-09-16T19:32:35.378 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2021-09-16T19:32:35.378 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2021-09-16T19:32:35.378 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:32:35.378 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2021-09-16T19:32:35.379 INFO:teuthology.orchestra.run.smithi118.stdout:====================================================================================================================== 2021-09-16T19:32:35.379 INFO:teuthology.orchestra.run.smithi118.stdout:Install 8 Packages 2021-09-16T19:32:35.379 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:32:35.380 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 36 M 2021-09-16T19:32:35.380 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 199 M 2021-09-16T19:32:35.380 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2021-09-16T19:32:35.445 INFO:teuthology.orchestra.run.smithi118.stdout:(1/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 55 MB/s | 3.7 MB 00:00 2021-09-16T19:32:35.500 INFO:teuthology.orchestra.run.smithi118.stdout:(2/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 88 MB/s | 4.7 MB 00:00 2021-09-16T19:32:35.649 INFO:teuthology.orchestra.run.smithi118.stdout:(3/8): python3-scipy-1.0.0-20.module+el8.1.0+33 54 MB/s | 15 MB 00:00 2021-09-16T19:32:35.663 INFO:teuthology.orchestra.run.smithi118.stdout:(4/8): openblas-0.3.12-1.el8.x86_64.rpm 28 MB/s | 4.6 MB 00:00 2021-09-16T19:32:35.687 INFO:teuthology.orchestra.run.smithi118.stdout:(5/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 5.9 MB/s | 225 kB 00:00 2021-09-16T19:32:35.694 INFO:teuthology.orchestra.run.smithi118.stdout:(6/8): libquadmath-8.4.1-1.el8.x86_64.rpm 6.1 MB/s | 169 kB 00:00 2021-09-16T19:32:35.711 INFO:teuthology.orchestra.run.smithi118.stdout:(7/8): libgfortran-8.4.1-1.el8.x86_64.rpm 25 MB/s | 639 kB 00:00 2021-09-16T19:32:35.887 INFO:teuthology.orchestra.run.smithi109.stdout: Preparing : 1/1 2021-09-16T19:32:35.905 INFO:teuthology.orchestra.run.smithi118.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-773 14 MB/s | 7.4 MB 00:00 2021-09-16T19:32:35.907 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:32:35.907 INFO:teuthology.orchestra.run.smithi118.stdout:Total 68 MB/s | 36 MB 00:00 2021-09-16T19:32:35.916 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2021-09-16T19:32:35.935 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2021-09-16T19:32:35.935 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2021-09-16T19:32:36.086 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-16T19:32:36.227 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-16T19:32:36.227 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2021-09-16T19:32:36.231 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2021-09-16T19:32:36.436 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-16T19:32:36.762 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2021-09-16T19:32:36.938 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-16T19:32:36.979 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-16T19:32:37.110 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-16T19:32:37.195 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-09-16T19:32:37.312 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-16T19:32:37.722 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-09-16T19:32:37.862 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-16T19:32:37.928 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2021-09-16T19:32:38.146 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-09-16T19:32:38.650 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2021-09-16T19:32:38.671 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-09-16T19:32:38.879 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2021-09-16T19:32:38.987 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-09-16T19:32:39.228 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-16T19:32:39.592 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2021-09-16T19:32:39.921 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-09-16T19:32:40.120 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-16T19:32:41.261 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-16T19:32:42.113 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-16T19:32:42.510 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2021-09-16T19:32:42.685 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-7730.gbc749ef 8/8 2021-09-16T19:32:42.969 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-7730.gbc749ef 8/8 2021-09-16T19:32:42.969 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-7730.gbc749ef 1/8 2021-09-16T19:32:42.969 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 2/8 2021-09-16T19:32:42.969 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 3/8 2021-09-16T19:32:42.970 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-09-16T19:32:42.970 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 5/8 2021-09-16T19:32:42.970 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-16T19:32:42.970 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 7/8 2021-09-16T19:32:43.298 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 8/8 2021-09-16T19:32:43.361 INFO:teuthology.orchestra.run.smithi109.stdout:Installed products updated. 2021-09-16T19:32:43.362 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2021-09-16T19:32:43.503 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-7730.gbc749ef 8/8 2021-09-16T19:32:43.765 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-7730.gbc749ef 8/8 2021-09-16T19:32:43.765 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-7730.gbc749ef 1/8 2021-09-16T19:32:43.765 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 2/8 2021-09-16T19:32:43.766 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 3/8 2021-09-16T19:32:43.766 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-09-16T19:32:43.766 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 5/8 2021-09-16T19:32:43.766 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-16T19:32:43.767 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 7/8 2021-09-16T19:32:44.083 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 8/8 2021-09-16T19:32:44.083 INFO:teuthology.orchestra.run.smithi118.stdout:Installed products updated. 2021-09-16T19:32:53.623 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:32:12 PM UTC. 2021-09-16T19:32:54.339 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:32:13 PM UTC. 2021-09-16T19:32:56.182 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:32:56.183 INFO:teuthology.orchestra.run.smithi109.stdout:Installed: 2021-09-16T19:32:56.183 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-mgr-diskprediction-local-2:17.0.0-7730.gbc749ef4.el8.noarch 2021-09-16T19:32:56.183 INFO:teuthology.orchestra.run.smithi109.stdout: libgfortran-8.4.1-1.el8.x86_64 2021-09-16T19:32:56.183 INFO:teuthology.orchestra.run.smithi109.stdout: libquadmath-8.4.1-1.el8.x86_64 2021-09-16T19:32:56.184 INFO:teuthology.orchestra.run.smithi109.stdout: openblas-0.3.12-1.el8.x86_64 2021-09-16T19:32:56.184 INFO:teuthology.orchestra.run.smithi109.stdout: openblas-threads-0.3.12-1.el8.x86_64 2021-09-16T19:32:56.184 INFO:teuthology.orchestra.run.smithi109.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-09-16T19:32:56.184 INFO:teuthology.orchestra.run.smithi109.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-09-16T19:32:56.184 INFO:teuthology.orchestra.run.smithi109.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2021-09-16T19:32:56.185 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:32:56.185 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:32:56.357 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install ceph-mgr-rook 2021-09-16T19:32:56.685 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:32:56.885 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:32:56.886 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2021-09-16T19:32:56.886 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local-2:17.0.0-7730.gbc749ef4.el8.noarch 2021-09-16T19:32:56.886 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran-8.4.1-1.el8.x86_64 2021-09-16T19:32:56.887 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath-8.4.1-1.el8.x86_64 2021-09-16T19:32:56.887 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-0.3.12-1.el8.x86_64 2021-09-16T19:32:56.887 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-threads-0.3.12-1.el8.x86_64 2021-09-16T19:32:56.888 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-09-16T19:32:56.888 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-09-16T19:32:56.888 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2021-09-16T19:32:56.888 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:32:56.889 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:32:57.046 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-rook 2021-09-16T19:32:57.184 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-09-16T19:32:57.376 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:32:57.885 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-09-16T19:33:01.996 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.4 kB 00:00 2021-09-16T19:33:02.700 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-09-16T19:33:07.267 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:04 ago on Thu 16 Sep 2021 07:28:26 PM UTC. 2021-09-16T19:33:08.009 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:28:27 PM UTC. 2021-09-16T19:33:09.852 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:33:09.853 INFO:teuthology.orchestra.run.smithi109.stdout:======================================================================================================== 2021-09-16T19:33:09.854 INFO:teuthology.orchestra.run.smithi109.stdout: Package Arch Version Repository Size 2021-09-16T19:33:09.854 INFO:teuthology.orchestra.run.smithi109.stdout:======================================================================================================== 2021-09-16T19:33:09.854 INFO:teuthology.orchestra.run.smithi109.stdout:Installing: 2021-09-16T19:33:09.854 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-mgr-rook noarch 2:17.0.0-7730.gbc749ef4.el8 ceph-noarch 45 k 2021-09-16T19:33:09.855 INFO:teuthology.orchestra.run.smithi109.stdout:Installing dependencies: 2021-09-16T19:33:09.855 INFO:teuthology.orchestra.run.smithi109.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-09-16T19:33:09.855 INFO:teuthology.orchestra.run.smithi109.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-09-16T19:33:09.855 INFO:teuthology.orchestra.run.smithi109.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-09-16T19:33:09.855 INFO:teuthology.orchestra.run.smithi109.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-09-16T19:33:09.856 INFO:teuthology.orchestra.run.smithi109.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2021-09-16T19:33:09.856 INFO:teuthology.orchestra.run.smithi109.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-09-16T19:33:09.856 INFO:teuthology.orchestra.run.smithi109.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2021-09-16T19:33:09.856 INFO:teuthology.orchestra.run.smithi109.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2021-09-16T19:33:09.857 INFO:teuthology.orchestra.run.smithi109.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2021-09-16T19:33:09.857 INFO:teuthology.orchestra.run.smithi109.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-09-16T19:33:09.857 INFO:teuthology.orchestra.run.smithi109.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-09-16T19:33:09.857 INFO:teuthology.orchestra.run.smithi109.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-09-16T19:33:09.857 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:33:09.858 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction Summary 2021-09-16T19:33:09.858 INFO:teuthology.orchestra.run.smithi109.stdout:======================================================================================================== 2021-09-16T19:33:09.858 INFO:teuthology.orchestra.run.smithi109.stdout:Install 13 Packages 2021-09-16T19:33:09.858 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:33:09.859 INFO:teuthology.orchestra.run.smithi109.stdout:Total download size: 1.9 M 2021-09-16T19:33:09.859 INFO:teuthology.orchestra.run.smithi109.stdout:Installed size: 25 M 2021-09-16T19:33:09.859 INFO:teuthology.orchestra.run.smithi109.stdout:Downloading Packages: 2021-09-16T19:33:09.932 INFO:teuthology.orchestra.run.smithi109.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 216 kB/s | 16 kB 00:00 2021-09-16T19:33:09.932 INFO:teuthology.orchestra.run.smithi109.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 434 kB/s | 33 kB 00:00 2021-09-16T19:33:09.983 INFO:teuthology.orchestra.run.smithi109.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 1.6 MB/s | 81 kB 00:00 2021-09-16T19:33:09.988 INFO:teuthology.orchestra.run.smithi109.stdout:(4/13): ceph-mgr-rook-17.0.0-7730.gbc749ef4.el8 339 kB/s | 45 kB 00:00 2021-09-16T19:33:10.009 INFO:teuthology.orchestra.run.smithi109.stdout:(5/13): python3-rsa-3.4.2-16.el8.noarch.rpm 2.7 MB/s | 73 kB 00:00 2021-09-16T19:33:10.044 INFO:teuthology.orchestra.run.smithi109.stdout:(6/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 759 kB/s | 27 kB 00:00 2021-09-16T19:33:10.067 INFO:teuthology.orchestra.run.smithi109.stdout:(7/13): python3-pyasn1-modules-0.3.7-6.el8.noar 4.9 MB/s | 110 kB 00:00 2021-09-16T19:33:10.087 INFO:teuthology.orchestra.run.smithi109.stdout:(8/13): python3-jsonpointer-1.10-11.el8.noarch. 957 kB/s | 20 kB 00:00 2021-09-16T19:33:10.102 INFO:teuthology.orchestra.run.smithi109.stdout:(9/13): python3-websocket-client-0.56.0-5.el8.n 539 kB/s | 61 kB 00:00 2021-09-16T19:33:10.111 INFO:teuthology.orchestra.run.smithi109.stdout:(10/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 5.4 MB/s | 126 kB 00:00 2021-09-16T19:33:10.134 INFO:teuthology.orchestra.run.smithi109.stdout:(11/13): python3-requests-oauthlib-1.0.0-1.el8. 1.3 MB/s | 43 kB 00:00 2021-09-16T19:33:10.139 INFO:teuthology.orchestra.run.smithi109.stdout:(12/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 5.4 MB/s | 155 kB 00:00 2021-09-16T19:33:10.160 INFO:teuthology.orchestra.run.smithi109.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.1 MB/s | 1.2 MB 00:00 2021-09-16T19:33:10.161 INFO:teuthology.orchestra.run.smithi109.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:33:10.161 INFO:teuthology.orchestra.run.smithi109.stdout:Total 6.3 MB/s | 1.9 MB 00:00 2021-09-16T19:33:10.166 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction check 2021-09-16T19:33:10.180 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction check succeeded. 2021-09-16T19:33:10.180 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction test 2021-09-16T19:33:10.247 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction test succeeded. 2021-09-16T19:33:10.251 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction 2021-09-16T19:33:10.550 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:33:10.552 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================================== 2021-09-16T19:33:10.552 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2021-09-16T19:33:10.552 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================================== 2021-09-16T19:33:10.552 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2021-09-16T19:33:10.553 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook noarch 2:17.0.0-7730.gbc749ef4.el8 ceph-noarch 45 k 2021-09-16T19:33:10.553 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2021-09-16T19:33:10.553 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-09-16T19:33:10.553 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-09-16T19:33:10.554 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-09-16T19:33:10.554 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-09-16T19:33:10.554 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2021-09-16T19:33:10.554 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-09-16T19:33:10.554 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2021-09-16T19:33:10.555 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2021-09-16T19:33:10.555 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2021-09-16T19:33:10.555 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-09-16T19:33:10.555 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-09-16T19:33:10.556 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-09-16T19:33:10.556 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:33:10.556 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2021-09-16T19:33:10.556 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================================== 2021-09-16T19:33:10.556 INFO:teuthology.orchestra.run.smithi118.stdout:Install 13 Packages 2021-09-16T19:33:10.557 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:33:10.557 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 1.9 M 2021-09-16T19:33:10.558 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 25 M 2021-09-16T19:33:10.558 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2021-09-16T19:33:10.634 INFO:teuthology.orchestra.run.smithi118.stdout:(1/13): python3-cachetools-3.1.1-4.el8.noarch.r 420 kB/s | 33 kB 00:00 2021-09-16T19:33:10.642 INFO:teuthology.orchestra.run.smithi118.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 187 kB/s | 16 kB 00:00 2021-09-16T19:33:10.687 INFO:teuthology.orchestra.run.smithi118.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 1.5 MB/s | 81 kB 00:00 2021-09-16T19:33:10.687 INFO:teuthology.orchestra.run.smithi118.stdout:(4/13): ceph-mgr-rook-17.0.0-7730.gbc749ef4.el8 338 kB/s | 45 kB 00:00 2021-09-16T19:33:10.715 INFO:teuthology.orchestra.run.smithi118.stdout:(5/13): python3-rsa-3.4.2-16.el8.noarch.rpm 2.5 MB/s | 73 kB 00:00 2021-09-16T19:33:10.747 INFO:teuthology.orchestra.run.smithi118.stdout:(6/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 821 kB/s | 27 kB 00:00 2021-09-16T19:33:10.768 INFO:teuthology.orchestra.run.smithi109.stdout: Preparing : 1/1 2021-09-16T19:33:10.768 INFO:teuthology.orchestra.run.smithi118.stdout:(7/13): python3-pyasn1-modules-0.3.7-6.el8.noar 5.1 MB/s | 110 kB 00:00 2021-09-16T19:33:10.788 INFO:teuthology.orchestra.run.smithi118.stdout:(8/13): python3-jsonpointer-1.10-11.el8.noarch. 1.0 MB/s | 20 kB 00:00 2021-09-16T19:33:10.803 INFO:teuthology.orchestra.run.smithi118.stdout:(9/13): python3-websocket-client-0.56.0-5.el8.n 529 kB/s | 61 kB 00:00 2021-09-16T19:33:10.812 INFO:teuthology.orchestra.run.smithi118.stdout:(10/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 5.1 MB/s | 126 kB 00:00 2021-09-16T19:33:10.834 INFO:teuthology.orchestra.run.smithi118.stdout:(11/13): python3-requests-oauthlib-1.0.0-1.el8. 1.3 MB/s | 43 kB 00:00 2021-09-16T19:33:10.839 INFO:teuthology.orchestra.run.smithi118.stdout:(12/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 5.6 MB/s | 155 kB 00:00 2021-09-16T19:33:10.918 INFO:teuthology.orchestra.run.smithi118.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.2 MB/s | 1.2 MB 00:00 2021-09-16T19:33:10.919 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:33:10.919 INFO:teuthology.orchestra.run.smithi118.stdout:Total 5.3 MB/s | 1.9 MB 00:00 2021-09-16T19:33:10.924 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2021-09-16T19:33:10.937 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2021-09-16T19:33:10.938 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2021-09-16T19:33:11.005 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2021-09-16T19:33:11.009 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2021-09-16T19:33:11.015 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-09-16T19:33:11.270 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 2/13 2021-09-16T19:33:11.536 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2021-09-16T19:33:11.547 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2021-09-16T19:33:11.779 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2021-09-16T19:33:11.788 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-09-16T19:33:11.986 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 2/13 2021-09-16T19:33:12.005 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2021-09-16T19:33:12.231 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2021-09-16T19:33:12.273 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2021-09-16T19:33:12.447 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2021-09-16T19:33:12.496 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2021-09-16T19:33:12.666 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2021-09-16T19:33:12.723 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2021-09-16T19:33:12.887 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-09-16T19:33:12.931 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2021-09-16T19:33:13.114 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2021-09-16T19:33:13.146 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2021-09-16T19:33:13.340 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2021-09-16T19:33:13.543 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-09-16T19:33:13.813 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2021-09-16T19:33:14.231 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2021-09-16T19:33:14.966 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2021-09-16T19:33:15.700 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2021-09-16T19:33:15.756 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2021-09-16T19:33:15.856 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-mgr-rook-2:17.0.0-7730.gbc749ef4.el8.noarch 13/13 2021-09-16T19:33:15.915 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-rook-2:17.0.0-7730.gbc749ef4.el8.noarch 13/13 2021-09-16T19:33:16.095 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-7730.gbc749ef4.el8.noarch 13/13 2021-09-16T19:33:16.095 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-mgr-rook-2:17.0.0-7730.gbc749ef4.el8.noarch 1/13 2021-09-16T19:33:16.096 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2021-09-16T19:33:16.096 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2021-09-16T19:33:16.096 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2021-09-16T19:33:16.096 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2021-09-16T19:33:16.096 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 6/13 2021-09-16T19:33:16.097 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2021-09-16T19:33:16.097 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 8/13 2021-09-16T19:33:16.097 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 9/13 2021-09-16T19:33:16.097 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 10/13 2021-09-16T19:33:16.098 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/13 2021-09-16T19:33:16.098 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 12/13 2021-09-16T19:33:16.152 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-7730.gbc749ef4.el8.noarch 13/13 2021-09-16T19:33:16.152 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-rook-2:17.0.0-7730.gbc749ef4.el8.noarch 1/13 2021-09-16T19:33:16.153 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2021-09-16T19:33:16.153 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2021-09-16T19:33:16.153 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2021-09-16T19:33:16.153 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2021-09-16T19:33:16.153 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 6/13 2021-09-16T19:33:16.154 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2021-09-16T19:33:16.154 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 8/13 2021-09-16T19:33:16.154 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 9/13 2021-09-16T19:33:16.154 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 10/13 2021-09-16T19:33:16.155 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/13 2021-09-16T19:33:16.155 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 12/13 2021-09-16T19:33:16.404 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 13/13 2021-09-16T19:33:16.405 INFO:teuthology.orchestra.run.smithi109.stdout:Installed products updated. 2021-09-16T19:33:16.517 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 13/13 2021-09-16T19:33:16.518 INFO:teuthology.orchestra.run.smithi118.stdout:Installed products updated. 2021-09-16T19:33:26.750 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:32:48 PM UTC. 2021-09-16T19:33:26.821 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:32:49 PM UTC. 2021-09-16T19:33:29.229 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:33:29.230 INFO:teuthology.orchestra.run.smithi109.stdout:Installed: 2021-09-16T19:33:29.230 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-mgr-rook-2:17.0.0-7730.gbc749ef4.el8.noarch 2021-09-16T19:33:29.230 INFO:teuthology.orchestra.run.smithi109.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-09-16T19:33:29.230 INFO:teuthology.orchestra.run.smithi109.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-09-16T19:33:29.231 INFO:teuthology.orchestra.run.smithi109.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-09-16T19:33:29.231 INFO:teuthology.orchestra.run.smithi109.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-09-16T19:33:29.231 INFO:teuthology.orchestra.run.smithi109.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-09-16T19:33:29.231 INFO:teuthology.orchestra.run.smithi109.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-09-16T19:33:29.232 INFO:teuthology.orchestra.run.smithi109.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-09-16T19:33:29.232 INFO:teuthology.orchestra.run.smithi109.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-09-16T19:33:29.232 INFO:teuthology.orchestra.run.smithi109.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-09-16T19:33:29.232 INFO:teuthology.orchestra.run.smithi109.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-09-16T19:33:29.232 INFO:teuthology.orchestra.run.smithi109.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-09-16T19:33:29.233 INFO:teuthology.orchestra.run.smithi109.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-09-16T19:33:29.233 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:33:29.233 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:33:29.327 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:33:29.327 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2021-09-16T19:33:29.328 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook-2:17.0.0-7730.gbc749ef4.el8.noarch 2021-09-16T19:33:29.328 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-09-16T19:33:29.328 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-09-16T19:33:29.328 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-09-16T19:33:29.329 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-09-16T19:33:29.329 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-09-16T19:33:29.329 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-09-16T19:33:29.329 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-09-16T19:33:29.330 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-09-16T19:33:29.330 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-09-16T19:33:29.331 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-09-16T19:33:29.331 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-09-16T19:33:29.331 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-09-16T19:33:29.332 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:33:29.332 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:33:29.368 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install ceph-mgr-cephadm 2021-09-16T19:33:29.457 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-cephadm 2021-09-16T19:33:29.701 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:33:29.789 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:33:30.206 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-09-16T19:33:30.284 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-09-16T19:33:35.056 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 2021-09-16T19:33:35.089 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-09-16T19:33:40.413 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:28:27 PM UTC. 2021-09-16T19:33:40.448 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:04 ago on Thu 16 Sep 2021 07:28:26 PM UTC. 2021-09-16T19:33:42.937 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:33:42.938 INFO:teuthology.orchestra.run.smithi109.stdout:=============================================================================================== 2021-09-16T19:33:42.939 INFO:teuthology.orchestra.run.smithi109.stdout: Package Arch Version Repository Size 2021-09-16T19:33:42.939 INFO:teuthology.orchestra.run.smithi109.stdout:=============================================================================================== 2021-09-16T19:33:42.939 INFO:teuthology.orchestra.run.smithi109.stdout:Installing: 2021-09-16T19:33:42.939 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-mgr-cephadm noarch 2:17.0.0-7730.gbc749ef4.el8 ceph-noarch 91 k 2021-09-16T19:33:42.939 INFO:teuthology.orchestra.run.smithi109.stdout:Installing dependencies: 2021-09-16T19:33:42.940 INFO:teuthology.orchestra.run.smithi109.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2021-09-16T19:33:42.940 INFO:teuthology.orchestra.run.smithi109.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2021-09-16T19:33:42.940 INFO:teuthology.orchestra.run.smithi109.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2021-09-16T19:33:42.940 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:33:42.941 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction Summary 2021-09-16T19:33:42.941 INFO:teuthology.orchestra.run.smithi109.stdout:=============================================================================================== 2021-09-16T19:33:42.941 INFO:teuthology.orchestra.run.smithi109.stdout:Install 4 Packages 2021-09-16T19:33:42.941 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:33:42.942 INFO:teuthology.orchestra.run.smithi109.stdout:Total download size: 5.9 M 2021-09-16T19:33:42.942 INFO:teuthology.orchestra.run.smithi109.stdout:Installed size: 25 M 2021-09-16T19:33:42.943 INFO:teuthology.orchestra.run.smithi109.stdout:Downloading Packages: 2021-09-16T19:33:42.975 INFO:teuthology.orchestra.run.smithi109.stdout:(1/4): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 15 MB/s | 538 kB 00:00 2021-09-16T19:33:42.987 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:33:42.988 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================================== 2021-09-16T19:33:42.988 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2021-09-16T19:33:42.988 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================================== 2021-09-16T19:33:42.989 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2021-09-16T19:33:42.989 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm noarch 2:17.0.0-7730.gbc749ef4.el8 ceph-noarch 91 k 2021-09-16T19:33:42.989 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2021-09-16T19:33:42.989 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2021-09-16T19:33:42.989 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2021-09-16T19:33:42.990 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2021-09-16T19:33:42.990 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:33:42.990 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2021-09-16T19:33:42.990 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================================== 2021-09-16T19:33:42.991 INFO:teuthology.orchestra.run.smithi118.stdout:Install 4 Packages 2021-09-16T19:33:42.991 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:33:42.992 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 5.9 M 2021-09-16T19:33:42.992 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 25 M 2021-09-16T19:33:42.992 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2021-09-16T19:33:43.022 INFO:teuthology.orchestra.run.smithi118.stdout:(1/4): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 16 MB/s | 538 kB 00:00 2021-09-16T19:33:43.023 INFO:teuthology.orchestra.run.smithi109.stdout:(2/4): python3-babel-2.5.1-5.el8.noarch.rpm 99 MB/s | 4.8 MB 00:00 2021-09-16T19:33:43.071 INFO:teuthology.orchestra.run.smithi118.stdout:(2/4): python3-babel-2.5.1-5.el8.noarch.rpm 99 MB/s | 4.8 MB 00:00 2021-09-16T19:33:43.107 INFO:teuthology.orchestra.run.smithi109.stdout:(3/4): ceph-mgr-cephadm-17.0.0-7730.gbc749ef4.e 550 kB/s | 91 kB 00:00 2021-09-16T19:33:43.154 INFO:teuthology.orchestra.run.smithi118.stdout:(3/4): ceph-mgr-cephadm-17.0.0-7730.gbc749ef4.e 558 kB/s | 91 kB 00:00 2021-09-16T19:33:43.242 INFO:teuthology.orchestra.run.smithi109.stdout:(4/4): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.5 MB/s | 476 kB 00:00 2021-09-16T19:33:43.244 INFO:teuthology.orchestra.run.smithi109.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:33:43.244 INFO:teuthology.orchestra.run.smithi109.stdout:Total 19 MB/s | 5.9 MB 00:00 2021-09-16T19:33:43.247 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction check 2021-09-16T19:33:43.256 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction check succeeded. 2021-09-16T19:33:43.256 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction test 2021-09-16T19:33:43.331 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction test succeeded. 2021-09-16T19:33:43.332 INFO:teuthology.orchestra.run.smithi118.stdout:(4/4): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.4 MB/s | 476 kB 00:00 2021-09-16T19:33:43.334 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:33:43.334 INFO:teuthology.orchestra.run.smithi118.stdout:Total 17 MB/s | 5.9 MB 00:00 2021-09-16T19:33:43.335 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction 2021-09-16T19:33:43.337 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2021-09-16T19:33:43.346 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2021-09-16T19:33:43.346 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2021-09-16T19:33:43.421 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2021-09-16T19:33:43.425 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2021-09-16T19:33:44.330 INFO:teuthology.orchestra.run.smithi109.stdout: Preparing : 1/1 2021-09-16T19:33:44.413 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2021-09-16T19:33:44.715 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/4 2021-09-16T19:33:44.798 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/4 2021-09-16T19:33:45.095 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/4 2021-09-16T19:33:45.183 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/4 2021-09-16T19:33:45.369 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 3/4 2021-09-16T19:33:45.475 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 3/4 2021-09-16T19:33:45.602 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 4/4 2021-09-16T19:33:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 4/4 2021-09-16T19:33:45.826 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 4/4 2021-09-16T19:33:45.827 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 1/4 2021-09-16T19:33:45.827 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/4 2021-09-16T19:33:45.827 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/4 2021-09-16T19:33:45.881 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 4/4 2021-09-16T19:33:45.881 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 1/4 2021-09-16T19:33:45.882 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/4 2021-09-16T19:33:45.882 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/4 2021-09-16T19:33:46.072 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/4 2021-09-16T19:33:46.072 INFO:teuthology.orchestra.run.smithi109.stdout:Installed products updated. 2021-09-16T19:33:46.180 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/4 2021-09-16T19:33:46.181 INFO:teuthology.orchestra.run.smithi118.stdout:Installed products updated. 2021-09-16T19:33:56.432 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:33:21 PM UTC. 2021-09-16T19:33:56.581 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:33:21 PM UTC. 2021-09-16T19:33:58.927 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:33:58.927 INFO:teuthology.orchestra.run.smithi109.stdout:Installed: 2021-09-16T19:33:58.927 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-mgr-cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 2021-09-16T19:33:58.928 INFO:teuthology.orchestra.run.smithi109.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2021-09-16T19:33:58.928 INFO:teuthology.orchestra.run.smithi109.stdout: python3-babel-2.5.1-5.el8.noarch 2021-09-16T19:33:58.928 INFO:teuthology.orchestra.run.smithi109.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-09-16T19:33:58.928 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:33:58.928 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:33:59.042 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:33:59.042 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2021-09-16T19:33:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm-2:17.0.0-7730.gbc749ef4.el8.noarch 2021-09-16T19:33:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2021-09-16T19:33:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel-2.5.1-5.el8.noarch 2021-09-16T19:33:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-09-16T19:33:59.044 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:33:59.044 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:33:59.056 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install ceph-fuse 2021-09-16T19:33:59.191 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-fuse 2021-09-16T19:33:59.394 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:33:59.527 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:33:59.894 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-09-16T19:34:00.017 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-09-16T19:34:04.726 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 77 kB/s | 2.4 kB 00:00 2021-09-16T19:34:04.797 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.4 kB 00:00 2021-09-16T19:34:10.020 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:28:26 PM UTC. 2021-09-16T19:34:10.080 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:04 ago on Thu 16 Sep 2021 07:28:27 PM UTC. 2021-09-16T19:34:12.518 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:34:12.520 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:34:12.520 INFO:teuthology.orchestra.run.smithi109.stdout: Package Arch Version Repository Size 2021-09-16T19:34:12.520 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:34:12.520 INFO:teuthology.orchestra.run.smithi109.stdout:Installing: 2021-09-16T19:34:12.520 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-fuse 2021-09-16T19:34:12.521 INFO:teuthology.orchestra.run.smithi109.stdout: x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 868 k 2021-09-16T19:34:12.521 INFO:teuthology.orchestra.run.smithi109.stdout:Installing dependencies: 2021-09-16T19:34:12.521 INFO:teuthology.orchestra.run.smithi109.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2021-09-16T19:34:12.521 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:34:12.522 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction Summary 2021-09-16T19:34:12.522 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:34:12.522 INFO:teuthology.orchestra.run.smithi109.stdout:Install 2 Packages 2021-09-16T19:34:12.522 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:34:12.523 INFO:teuthology.orchestra.run.smithi109.stdout:Total download size: 950 k 2021-09-16T19:34:12.523 INFO:teuthology.orchestra.run.smithi109.stdout:Installed size: 3.1 M 2021-09-16T19:34:12.523 INFO:teuthology.orchestra.run.smithi109.stdout:Downloading Packages: 2021-09-16T19:34:12.604 INFO:teuthology.orchestra.run.smithi109.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 989 kB/s | 83 kB 00:00 2021-09-16T19:34:12.622 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:34:12.622 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:34:12.623 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2021-09-16T19:34:12.623 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:34:12.623 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2021-09-16T19:34:12.623 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse 2021-09-16T19:34:12.624 INFO:teuthology.orchestra.run.smithi118.stdout: x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 868 k 2021-09-16T19:34:12.624 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2021-09-16T19:34:12.624 INFO:teuthology.orchestra.run.smithi118.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2021-09-16T19:34:12.624 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:34:12.625 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2021-09-16T19:34:12.625 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:34:12.625 INFO:teuthology.orchestra.run.smithi118.stdout:Install 2 Packages 2021-09-16T19:34:12.625 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:34:12.626 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 950 k 2021-09-16T19:34:12.626 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 3.1 M 2021-09-16T19:34:12.626 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2021-09-16T19:34:12.658 INFO:teuthology.orchestra.run.smithi118.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.4 MB/s | 83 kB 00:00 2021-09-16T19:34:12.790 INFO:teuthology.orchestra.run.smithi109.stdout:(2/2): ceph-fuse-17.0.0-7730.gbc749ef4.el8.x86_ 3.1 MB/s | 868 kB 00:00 2021-09-16T19:34:12.791 INFO:teuthology.orchestra.run.smithi109.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:34:12.791 INFO:teuthology.orchestra.run.smithi109.stdout:Total 3.4 MB/s | 950 kB 00:00 2021-09-16T19:34:12.792 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction check 2021-09-16T19:34:12.813 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction check succeeded. 2021-09-16T19:34:12.813 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction test 2021-09-16T19:34:12.841 INFO:teuthology.orchestra.run.smithi118.stdout:(2/2): ceph-fuse-17.0.0-7730.gbc749ef4.el8.x86_ 3.9 MB/s | 868 kB 00:00 2021-09-16T19:34:12.842 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:34:12.843 INFO:teuthology.orchestra.run.smithi118.stdout:Total 4.2 MB/s | 950 kB 00:00 2021-09-16T19:34:12.843 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2021-09-16T19:34:12.864 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2021-09-16T19:34:12.865 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2021-09-16T19:34:12.885 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction test succeeded. 2021-09-16T19:34:12.889 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction 2021-09-16T19:34:12.938 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2021-09-16T19:34:12.942 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2021-09-16T19:34:13.082 INFO:teuthology.orchestra.run.smithi109.stdout: Preparing : 1/1 2021-09-16T19:34:13.131 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2021-09-16T19:34:13.391 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-09-16T19:34:13.439 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-09-16T19:34:13.564 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : ceph-fuse-2:17.0.0-7730.gbc749ef4.el8.x86_64 2/2 2021-09-16T19:34:13.612 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-fuse-2:17.0.0-7730.gbc749ef4.el8.x86_64 2/2 2021-09-16T19:34:13.879 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: ceph-fuse-2:17.0.0-7730.gbc749ef4.el8.x86_64 2/2 2021-09-16T19:34:13.879 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : ceph-fuse-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/2 2021-09-16T19:34:13.945 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-fuse-2:17.0.0-7730.gbc749ef4.el8.x86_64 2/2 2021-09-16T19:34:13.945 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-fuse-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/2 2021-09-16T19:34:14.143 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-09-16T19:34:14.143 INFO:teuthology.orchestra.run.smithi109.stdout:Installed products updated. 2021-09-16T19:34:14.197 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-09-16T19:34:14.197 INFO:teuthology.orchestra.run.smithi118.stdout:Installed products updated. 2021-09-16T19:34:24.440 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:33:51 PM UTC. 2021-09-16T19:34:24.672 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:33:51 PM UTC. 2021-09-16T19:34:26.932 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:34:26.932 INFO:teuthology.orchestra.run.smithi109.stdout:Installed: 2021-09-16T19:34:26.933 INFO:teuthology.orchestra.run.smithi109.stdout: ceph-fuse-2:17.0.0-7730.gbc749ef4.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-09-16T19:34:26.933 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:34:26.933 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:34:27.067 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install librados-devel 2021-09-16T19:34:27.186 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:34:27.186 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2021-09-16T19:34:27.187 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse-2:17.0.0-7730.gbc749ef4.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-09-16T19:34:27.187 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:34:27.187 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:34:27.350 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados-devel 2021-09-16T19:34:27.403 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:34:27.690 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:34:27.912 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-09-16T19:34:28.181 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 95 kB/s | 2.8 kB 00:00 2021-09-16T19:34:32.989 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-09-16T19:34:32.999 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 76 kB/s | 2.4 kB 00:00 2021-09-16T19:34:38.310 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:28:26 PM UTC. 2021-09-16T19:34:38.523 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:04 ago on Thu 16 Sep 2021 07:28:27 PM UTC. 2021-09-16T19:34:40.815 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:34:40.816 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:34:40.816 INFO:teuthology.orchestra.run.smithi109.stdout: Package Arch Version Repo Size 2021-09-16T19:34:40.817 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:34:40.817 INFO:teuthology.orchestra.run.smithi109.stdout:Installing: 2021-09-16T19:34:40.817 INFO:teuthology.orchestra.run.smithi109.stdout: librados-devel x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 125 k 2021-09-16T19:34:40.817 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:34:40.818 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction Summary 2021-09-16T19:34:40.818 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:34:40.818 INFO:teuthology.orchestra.run.smithi109.stdout:Install 1 Package 2021-09-16T19:34:40.818 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:34:40.819 INFO:teuthology.orchestra.run.smithi109.stdout:Total download size: 125 k 2021-09-16T19:34:40.819 INFO:teuthology.orchestra.run.smithi109.stdout:Installed size: 432 k 2021-09-16T19:34:40.819 INFO:teuthology.orchestra.run.smithi109.stdout:Downloading Packages: 2021-09-16T19:34:41.047 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:34:41.048 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:34:41.048 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2021-09-16T19:34:41.048 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:34:41.048 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2021-09-16T19:34:41.049 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 125 k 2021-09-16T19:34:41.049 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:34:41.049 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2021-09-16T19:34:41.049 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:34:41.050 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2021-09-16T19:34:41.050 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:34:41.050 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 125 k 2021-09-16T19:34:41.050 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 432 k 2021-09-16T19:34:41.051 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2021-09-16T19:34:41.319 INFO:teuthology.orchestra.run.smithi109.stdout:librados-devel-17.0.0-7730.gbc749ef4.el8.x86_64 249 kB/s | 125 kB 00:00 2021-09-16T19:34:41.320 INFO:teuthology.orchestra.run.smithi109.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:34:41.320 INFO:teuthology.orchestra.run.smithi109.stdout:Total 248 kB/s | 125 kB 00:00 2021-09-16T19:34:41.320 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction check 2021-09-16T19:34:41.327 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction check succeeded. 2021-09-16T19:34:41.327 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction test 2021-09-16T19:34:41.383 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction test succeeded. 2021-09-16T19:34:41.387 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction 2021-09-16T19:34:41.500 INFO:teuthology.orchestra.run.smithi109.stdout: Preparing : 1/1 2021-09-16T19:34:41.530 INFO:teuthology.orchestra.run.smithi118.stdout:librados-devel-17.0.0-7730.gbc749ef4.el8.x86_64 259 kB/s | 125 kB 00:00 2021-09-16T19:34:41.531 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:34:41.531 INFO:teuthology.orchestra.run.smithi118.stdout:Total 258 kB/s | 125 kB 00:00 2021-09-16T19:34:41.532 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2021-09-16T19:34:41.538 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2021-09-16T19:34:41.538 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2021-09-16T19:34:41.594 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2021-09-16T19:34:41.598 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2021-09-16T19:34:41.715 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2021-09-16T19:34:41.726 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : librados-devel-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:34:41.898 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librados-devel-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:34:41.941 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: librados-devel-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:34:42.159 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados-devel-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:34:42.176 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : librados-devel-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:34:42.177 INFO:teuthology.orchestra.run.smithi109.stdout:Installed products updated. 2021-09-16T19:34:42.419 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados-devel-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:34:42.419 INFO:teuthology.orchestra.run.smithi118.stdout:Installed products updated. 2021-09-16T19:34:52.579 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:34:19 PM UTC. 2021-09-16T19:34:52.704 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:34:19 PM UTC. 2021-09-16T19:34:55.049 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:34:55.049 INFO:teuthology.orchestra.run.smithi109.stdout:Installed: 2021-09-16T19:34:55.049 INFO:teuthology.orchestra.run.smithi109.stdout: librados-devel-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:34:55.050 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:34:55.050 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:34:55.170 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install libcephfs2 2021-09-16T19:34:55.189 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:34:55.189 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2021-09-16T19:34:55.189 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:34:55.190 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:34:55.190 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:34:55.346 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs2 2021-09-16T19:34:55.507 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:34:55.683 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:34:56.035 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 82 kB/s | 2.8 kB 00:00 2021-09-16T19:34:56.159 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-09-16T19:35:00.924 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 85 kB/s | 2.4 kB 00:00 2021-09-16T19:35:01.049 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 85 kB/s | 2.4 kB 00:00 2021-09-16T19:35:06.373 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:04 ago on Thu 16 Sep 2021 07:28:26 PM UTC. 2021-09-16T19:35:06.430 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:04 ago on Thu 16 Sep 2021 07:28:27 PM UTC. 2021-09-16T19:35:08.491 INFO:teuthology.orchestra.run.smithi118.stdout:Package libcephfs2-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:35:08.521 INFO:teuthology.orchestra.run.smithi109.stdout:Package libcephfs2-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:35:08.926 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:35:08.927 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2021-09-16T19:35:08.928 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:35:08.948 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:35:08.949 INFO:teuthology.orchestra.run.smithi109.stdout:Nothing to do. 2021-09-16T19:35:08.949 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:35:09.037 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs-devel 2021-09-16T19:35:09.059 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install libcephfs-devel 2021-09-16T19:35:09.375 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:35:09.394 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:35:09.866 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-09-16T19:35:09.889 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 100 kB/s | 2.8 kB 00:00 2021-09-16T19:35:10.043 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 85 kB/s | 2.4 kB 00:00 2021-09-16T19:35:10.066 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-09-16T19:35:11.835 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:35:11.836 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:35:11.836 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2021-09-16T19:35:11.836 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:35:11.836 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2021-09-16T19:35:11.837 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 24 k 2021-09-16T19:35:11.837 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:35:11.837 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2021-09-16T19:35:11.837 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:35:11.837 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2021-09-16T19:35:11.838 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:35:11.838 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 24 k 2021-09-16T19:35:11.838 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 102 k 2021-09-16T19:35:11.838 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2021-09-16T19:35:11.917 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:35:11.919 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:35:11.919 INFO:teuthology.orchestra.run.smithi109.stdout: Package Arch Version Repo Size 2021-09-16T19:35:11.919 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:35:11.919 INFO:teuthology.orchestra.run.smithi109.stdout:Installing: 2021-09-16T19:35:11.920 INFO:teuthology.orchestra.run.smithi109.stdout: libcephfs-devel x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 24 k 2021-09-16T19:35:11.920 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:35:11.920 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction Summary 2021-09-16T19:35:11.920 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:35:11.920 INFO:teuthology.orchestra.run.smithi109.stdout:Install 1 Package 2021-09-16T19:35:11.921 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:35:11.921 INFO:teuthology.orchestra.run.smithi109.stdout:Total download size: 24 k 2021-09-16T19:35:11.921 INFO:teuthology.orchestra.run.smithi109.stdout:Installed size: 102 k 2021-09-16T19:35:11.921 INFO:teuthology.orchestra.run.smithi109.stdout:Downloading Packages: 2021-09-16T19:35:12.190 INFO:teuthology.orchestra.run.smithi118.stdout:libcephfs-devel-17.0.0-7730.gbc749ef4.el8.x86_6 68 kB/s | 24 kB 00:00 2021-09-16T19:35:12.191 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:35:12.192 INFO:teuthology.orchestra.run.smithi118.stdout:Total 67 kB/s | 24 kB 00:00 2021-09-16T19:35:12.192 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2021-09-16T19:35:12.195 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2021-09-16T19:35:12.196 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2021-09-16T19:35:12.197 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2021-09-16T19:35:12.202 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2021-09-16T19:35:12.290 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2021-09-16T19:35:12.294 INFO:teuthology.orchestra.run.smithi109.stdout:libcephfs-devel-17.0.0-7730.gbc749ef4.el8.x86_6 64 kB/s | 24 kB 00:00 2021-09-16T19:35:12.295 INFO:teuthology.orchestra.run.smithi109.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:35:12.296 INFO:teuthology.orchestra.run.smithi109.stdout:Total 63 kB/s | 24 kB 00:00 2021-09-16T19:35:12.296 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction check 2021-09-16T19:35:12.299 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction check succeeded. 2021-09-16T19:35:12.300 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction test 2021-09-16T19:35:12.301 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction test succeeded. 2021-09-16T19:35:12.305 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction 2021-09-16T19:35:12.358 INFO:teuthology.orchestra.run.smithi109.stdout: Preparing : 1/1 2021-09-16T19:35:12.427 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs-devel-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:35:12.555 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : libcephfs-devel-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:35:12.646 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:35:12.749 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:35:12.866 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs-devel-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:35:12.866 INFO:teuthology.orchestra.run.smithi118.stdout:Installed products updated. 2021-09-16T19:35:12.968 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : libcephfs-devel-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:35:12.968 INFO:teuthology.orchestra.run.smithi109.stdout:Installed products updated. 2021-09-16T19:35:23.176 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:04 ago on Thu 16 Sep 2021 07:34:47 PM UTC. 2021-09-16T19:35:23.310 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:04 ago on Thu 16 Sep 2021 07:34:47 PM UTC. 2021-09-16T19:35:25.673 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:35:25.674 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2021-09-16T19:35:25.674 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:35:25.674 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:35:25.675 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:35:25.804 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados2 2021-09-16T19:35:25.839 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:35:25.839 INFO:teuthology.orchestra.run.smithi109.stdout:Installed: 2021-09-16T19:35:25.839 INFO:teuthology.orchestra.run.smithi109.stdout: libcephfs-devel-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:35:25.840 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:35:25.840 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:35:25.975 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install librados2 2021-09-16T19:35:26.139 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:35:26.312 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:35:26.631 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-09-16T19:35:26.802 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-09-16T19:35:31.638 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-09-16T19:35:31.725 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 83 kB/s | 2.4 kB 00:00 2021-09-16T19:35:37.006 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:28:27 PM UTC. 2021-09-16T19:35:37.166 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:28:26 PM UTC. 2021-09-16T19:35:39.077 INFO:teuthology.orchestra.run.smithi118.stdout:Package librados2-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:35:39.283 INFO:teuthology.orchestra.run.smithi109.stdout:Package librados2-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:35:39.498 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:35:39.500 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2021-09-16T19:35:39.500 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:35:39.607 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librbd1 2021-09-16T19:35:39.712 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:35:39.713 INFO:teuthology.orchestra.run.smithi109.stdout:Nothing to do. 2021-09-16T19:35:39.713 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:35:39.821 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install librbd1 2021-09-16T19:35:39.941 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:35:40.158 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:35:40.456 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-09-16T19:35:40.630 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-09-16T19:35:40.639 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 99 kB/s | 2.8 kB 00:00 2021-09-16T19:35:40.810 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 88 kB/s | 2.4 kB 00:00 2021-09-16T19:35:42.019 INFO:teuthology.orchestra.run.smithi118.stdout:Package librbd1-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:35:42.183 INFO:teuthology.orchestra.run.smithi109.stdout:Package librbd1-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:35:42.441 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:35:42.442 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2021-09-16T19:35:42.442 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:35:42.537 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rados 2021-09-16T19:35:42.609 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:35:42.610 INFO:teuthology.orchestra.run.smithi109.stdout:Nothing to do. 2021-09-16T19:35:42.611 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:35:42.713 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install python3-rados 2021-09-16T19:35:42.873 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:35:43.049 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:35:43.365 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-09-16T19:35:43.540 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 91 kB/s | 2.8 kB 00:00 2021-09-16T19:35:43.542 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.4 kB 00:00 2021-09-16T19:35:43.718 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 77 kB/s | 2.4 kB 00:00 2021-09-16T19:35:44.976 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rados-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:35:45.122 INFO:teuthology.orchestra.run.smithi109.stdout:Package python3-rados-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:35:45.402 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:35:45.403 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2021-09-16T19:35:45.403 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:35:45.511 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rgw 2021-09-16T19:35:45.546 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:35:45.547 INFO:teuthology.orchestra.run.smithi109.stdout:Nothing to do. 2021-09-16T19:35:45.547 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:35:45.690 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install python3-rgw 2021-09-16T19:35:45.847 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:35:46.025 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:35:46.371 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 97 kB/s | 2.8 kB 00:00 2021-09-16T19:35:46.499 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-09-16T19:35:46.547 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-09-16T19:35:46.673 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-09-16T19:35:47.926 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rgw-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:35:48.116 INFO:teuthology.orchestra.run.smithi109.stdout:Package python3-rgw-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:35:48.355 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:35:48.356 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2021-09-16T19:35:48.356 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:35:48.461 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-cephfs 2021-09-16T19:35:48.539 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:35:48.540 INFO:teuthology.orchestra.run.smithi109.stdout:Nothing to do. 2021-09-16T19:35:48.540 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:35:48.649 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install python3-cephfs 2021-09-16T19:35:48.796 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:35:48.983 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:35:49.277 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-09-16T19:35:49.449 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 86 kB/s | 2.4 kB 00:00 2021-09-16T19:35:49.459 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 102 kB/s | 2.8 kB 00:00 2021-09-16T19:35:49.630 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 85 kB/s | 2.4 kB 00:00 2021-09-16T19:35:50.850 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-cephfs-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:35:51.017 INFO:teuthology.orchestra.run.smithi109.stdout:Package python3-cephfs-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:35:51.276 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:35:51.277 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2021-09-16T19:35:51.277 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:35:51.414 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rbd 2021-09-16T19:35:51.444 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:35:51.445 INFO:teuthology.orchestra.run.smithi109.stdout:Nothing to do. 2021-09-16T19:35:51.445 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:35:51.563 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install python3-rbd 2021-09-16T19:35:51.750 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:35:51.901 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:35:52.233 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-09-16T19:35:52.385 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 2021-09-16T19:35:52.408 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 85 kB/s | 2.4 kB 00:00 2021-09-16T19:35:52.558 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 82 kB/s | 2.4 kB 00:00 2021-09-16T19:35:53.792 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rbd-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:35:53.982 INFO:teuthology.orchestra.run.smithi109.stdout:Package python3-rbd-2:17.0.0-7730.gbc749ef4.el8.x86_64 is already installed. 2021-09-16T19:35:54.217 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:35:54.219 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2021-09-16T19:35:54.219 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:35:54.320 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-fuse 2021-09-16T19:35:54.405 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:35:54.406 INFO:teuthology.orchestra.run.smithi109.stdout:Nothing to do. 2021-09-16T19:35:54.406 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:35:54.528 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install rbd-fuse 2021-09-16T19:35:54.654 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:35:54.862 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:35:55.161 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 98 kB/s | 2.8 kB 00:00 2021-09-16T19:35:55.335 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 84 kB/s | 2.4 kB 00:00 2021-09-16T19:35:55.355 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 99 kB/s | 2.8 kB 00:00 2021-09-16T19:35:55.527 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 85 kB/s | 2.4 kB 00:00 2021-09-16T19:35:57.149 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:35:57.151 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:35:57.151 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2021-09-16T19:35:57.152 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:35:57.152 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2021-09-16T19:35:57.152 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 93 k 2021-09-16T19:35:57.153 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:35:57.153 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2021-09-16T19:35:57.154 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:35:57.154 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2021-09-16T19:35:57.155 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:35:57.155 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 93 k 2021-09-16T19:35:57.155 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 281 k 2021-09-16T19:35:57.156 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2021-09-16T19:35:57.305 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:35:57.307 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:35:57.307 INFO:teuthology.orchestra.run.smithi109.stdout: Package Architecture Version Repository Size 2021-09-16T19:35:57.308 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:35:57.308 INFO:teuthology.orchestra.run.smithi109.stdout:Installing: 2021-09-16T19:35:57.308 INFO:teuthology.orchestra.run.smithi109.stdout: rbd-fuse x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 93 k 2021-09-16T19:35:57.309 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:35:57.309 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction Summary 2021-09-16T19:35:57.310 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:35:57.310 INFO:teuthology.orchestra.run.smithi109.stdout:Install 1 Package 2021-09-16T19:35:57.311 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:35:57.311 INFO:teuthology.orchestra.run.smithi109.stdout:Total download size: 93 k 2021-09-16T19:35:57.311 INFO:teuthology.orchestra.run.smithi109.stdout:Installed size: 281 k 2021-09-16T19:35:57.311 INFO:teuthology.orchestra.run.smithi109.stdout:Downloading Packages: 2021-09-16T19:35:57.568 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-fuse-17.0.0-7730.gbc749ef4.el8.x86_64.rpm 223 kB/s | 93 kB 00:00 2021-09-16T19:35:57.569 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:35:57.570 INFO:teuthology.orchestra.run.smithi118.stdout:Total 222 kB/s | 93 kB 00:00 2021-09-16T19:35:57.570 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2021-09-16T19:35:57.581 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2021-09-16T19:35:57.582 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2021-09-16T19:35:57.636 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2021-09-16T19:35:57.640 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2021-09-16T19:35:57.755 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2021-09-16T19:35:57.764 INFO:teuthology.orchestra.run.smithi109.stdout:rbd-fuse-17.0.0-7730.gbc749ef4.el8.x86_64.rpm 204 kB/s | 93 kB 00:00 2021-09-16T19:35:57.765 INFO:teuthology.orchestra.run.smithi109.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:35:57.766 INFO:teuthology.orchestra.run.smithi109.stdout:Total 203 kB/s | 93 kB 00:00 2021-09-16T19:35:57.766 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction check 2021-09-16T19:35:57.777 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction check succeeded. 2021-09-16T19:35:57.777 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction test 2021-09-16T19:35:57.831 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction test succeeded. 2021-09-16T19:35:57.835 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction 2021-09-16T19:35:57.959 INFO:teuthology.orchestra.run.smithi109.stdout: Preparing : 1/1 2021-09-16T19:35:57.961 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-fuse-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:35:58.141 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : rbd-fuse-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:35:58.168 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-fuse-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:35:58.370 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: rbd-fuse-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:35:58.397 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-fuse-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:35:58.398 INFO:teuthology.orchestra.run.smithi118.stdout:Installed products updated. 2021-09-16T19:35:58.586 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : rbd-fuse-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:35:58.586 INFO:teuthology.orchestra.run.smithi109.stdout:Installed products updated. 2021-09-16T19:36:08.691 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:35:18 PM UTC. 2021-09-16T19:36:08.886 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:35:18 PM UTC. 2021-09-16T19:36:11.191 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:36:11.191 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2021-09-16T19:36:11.193 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:36:11.194 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:36:11.194 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:36:11.321 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-mirror 2021-09-16T19:36:11.448 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:36:11.449 INFO:teuthology.orchestra.run.smithi109.stdout:Installed: 2021-09-16T19:36:11.449 INFO:teuthology.orchestra.run.smithi109.stdout: rbd-fuse-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:36:11.449 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:36:11.450 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:36:11.572 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install rbd-mirror 2021-09-16T19:36:11.655 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:36:11.908 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:36:12.144 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 94 kB/s | 2.8 kB 00:00 2021-09-16T19:36:12.400 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 90 kB/s | 2.8 kB 00:00 2021-09-16T19:36:17.045 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.4 kB 00:00 2021-09-16T19:36:17.307 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 86 kB/s | 2.4 kB 00:00 2021-09-16T19:36:22.325 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:04 ago on Thu 16 Sep 2021 07:28:27 PM UTC. 2021-09-16T19:36:22.517 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:28:26 PM UTC. 2021-09-16T19:36:24.816 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:36:24.817 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:36:24.817 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2021-09-16T19:36:24.817 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:36:24.817 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2021-09-16T19:36:24.818 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 3.8 M 2021-09-16T19:36:24.818 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:36:24.818 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2021-09-16T19:36:24.818 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:36:24.819 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2021-09-16T19:36:24.819 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:36:24.819 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 3.8 M 2021-09-16T19:36:24.819 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 19 M 2021-09-16T19:36:24.819 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2021-09-16T19:36:25.008 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:36:25.009 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:36:25.010 INFO:teuthology.orchestra.run.smithi109.stdout: Package Arch Version Repo Size 2021-09-16T19:36:25.010 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:36:25.010 INFO:teuthology.orchestra.run.smithi109.stdout:Installing: 2021-09-16T19:36:25.010 INFO:teuthology.orchestra.run.smithi109.stdout: rbd-mirror x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 3.8 M 2021-09-16T19:36:25.011 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:36:25.011 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction Summary 2021-09-16T19:36:25.011 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:36:25.011 INFO:teuthology.orchestra.run.smithi109.stdout:Install 1 Package 2021-09-16T19:36:25.011 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:36:25.012 INFO:teuthology.orchestra.run.smithi109.stdout:Total download size: 3.8 M 2021-09-16T19:36:25.012 INFO:teuthology.orchestra.run.smithi109.stdout:Installed size: 19 M 2021-09-16T19:36:25.012 INFO:teuthology.orchestra.run.smithi109.stdout:Downloading Packages: 2021-09-16T19:36:25.143 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-mirror-17.0.0-7730.gbc749ef4.el8.x86_64.rpm 12 MB/s | 3.8 MB 00:00 2021-09-16T19:36:25.144 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:36:25.144 INFO:teuthology.orchestra.run.smithi118.stdout:Total 11 MB/s | 3.8 MB 00:00 2021-09-16T19:36:25.145 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2021-09-16T19:36:25.158 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2021-09-16T19:36:25.158 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2021-09-16T19:36:25.227 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2021-09-16T19:36:25.231 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2021-09-16T19:36:25.341 INFO:teuthology.orchestra.run.smithi109.stdout:rbd-mirror-17.0.0-7730.gbc749ef4.el8.x86_64.rpm 11 MB/s | 3.8 MB 00:00 2021-09-16T19:36:25.343 INFO:teuthology.orchestra.run.smithi109.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:36:25.343 INFO:teuthology.orchestra.run.smithi109.stdout:Total 11 MB/s | 3.8 MB 00:00 2021-09-16T19:36:25.343 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction check 2021-09-16T19:36:25.356 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction check succeeded. 2021-09-16T19:36:25.356 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction test 2021-09-16T19:36:25.424 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction test succeeded. 2021-09-16T19:36:25.429 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction 2021-09-16T19:36:25.713 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2021-09-16T19:36:25.909 INFO:teuthology.orchestra.run.smithi109.stdout: Preparing : 1/1 2021-09-16T19:36:25.960 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-mirror-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:36:26.132 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : rbd-mirror-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:36:26.322 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-mirror-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:36:26.484 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: rbd-mirror-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:36:26.565 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-mirror-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:36:26.566 INFO:teuthology.orchestra.run.smithi118.stdout:Installed products updated. 2021-09-16T19:36:26.759 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : rbd-mirror-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:36:26.759 INFO:teuthology.orchestra.run.smithi109.stdout:Installed products updated. 2021-09-16T19:36:36.990 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:36:03 PM UTC. 2021-09-16T19:36:37.176 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:04 ago on Thu 16 Sep 2021 07:36:03 PM UTC. 2021-09-16T19:36:39.493 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:36:39.493 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2021-09-16T19:36:39.493 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:36:39.494 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:36:39.494 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:36:39.629 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-nbd 2021-09-16T19:36:39.710 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:36:39.711 INFO:teuthology.orchestra.run.smithi109.stdout:Installed: 2021-09-16T19:36:39.711 INFO:teuthology.orchestra.run.smithi109.stdout: rbd-mirror-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:36:39.711 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:36:39.712 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:36:39.835 DEBUG:teuthology.orchestra.run.smithi109:> sudo yum -y install rbd-nbd 2021-09-16T19:36:40.025 INFO:teuthology.orchestra.run.smithi118.stdout:Updating Subscription Management repositories. 2021-09-16T19:36:40.180 INFO:teuthology.orchestra.run.smithi109.stdout:Updating Subscription Management repositories. 2021-09-16T19:36:40.537 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 93 kB/s | 2.8 kB 00:00 2021-09-16T19:36:40.701 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 99 kB/s | 2.8 kB 00:00 2021-09-16T19:36:45.354 INFO:teuthology.orchestra.run.smithi118.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 2021-09-16T19:36:45.502 INFO:teuthology.orchestra.run.smithi109.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 85 kB/s | 2.4 kB 00:00 2021-09-16T19:36:50.652 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:28:27 PM UTC. 2021-09-16T19:36:50.885 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:28:26 PM UTC. 2021-09-16T19:36:53.195 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2021-09-16T19:36:53.196 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:36:53.196 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2021-09-16T19:36:53.196 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:36:53.197 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2021-09-16T19:36:53.197 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 173 k 2021-09-16T19:36:53.197 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:36:53.197 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2021-09-16T19:36:53.198 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2021-09-16T19:36:53.198 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2021-09-16T19:36:53.198 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:36:53.198 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 173 k 2021-09-16T19:36:53.199 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 563 k 2021-09-16T19:36:53.199 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2021-09-16T19:36:53.367 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-nbd-17.0.0-7730.gbc749ef4.el8.x86_64.rpm 1.0 MB/s | 173 kB 00:00 2021-09-16T19:36:53.368 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:36:53.368 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.0 MB/s | 173 kB 00:00 2021-09-16T19:36:53.368 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2021-09-16T19:36:53.380 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2021-09-16T19:36:53.381 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2021-09-16T19:36:53.435 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2021-09-16T19:36:53.439 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2021-09-16T19:36:53.481 INFO:teuthology.orchestra.run.smithi109.stdout:Dependencies resolved. 2021-09-16T19:36:53.482 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:36:53.482 INFO:teuthology.orchestra.run.smithi109.stdout: Package Architecture Version Repository Size 2021-09-16T19:36:53.482 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:36:53.483 INFO:teuthology.orchestra.run.smithi109.stdout:Installing: 2021-09-16T19:36:53.483 INFO:teuthology.orchestra.run.smithi109.stdout: rbd-nbd x86_64 2:17.0.0-7730.gbc749ef4.el8 ceph 173 k 2021-09-16T19:36:53.483 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:36:53.483 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction Summary 2021-09-16T19:36:53.484 INFO:teuthology.orchestra.run.smithi109.stdout:================================================================================ 2021-09-16T19:36:53.484 INFO:teuthology.orchestra.run.smithi109.stdout:Install 1 Package 2021-09-16T19:36:53.484 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:36:53.484 INFO:teuthology.orchestra.run.smithi109.stdout:Total download size: 173 k 2021-09-16T19:36:53.484 INFO:teuthology.orchestra.run.smithi109.stdout:Installed size: 563 k 2021-09-16T19:36:53.485 INFO:teuthology.orchestra.run.smithi109.stdout:Downloading Packages: 2021-09-16T19:36:53.557 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2021-09-16T19:36:53.634 INFO:teuthology.orchestra.run.smithi109.stdout:rbd-nbd-17.0.0-7730.gbc749ef4.el8.x86_64.rpm 1.1 MB/s | 173 kB 00:00 2021-09-16T19:36:53.635 INFO:teuthology.orchestra.run.smithi109.stdout:-------------------------------------------------------------------------------- 2021-09-16T19:36:53.636 INFO:teuthology.orchestra.run.smithi109.stdout:Total 1.1 MB/s | 173 kB 00:00 2021-09-16T19:36:53.636 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction check 2021-09-16T19:36:53.648 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction check succeeded. 2021-09-16T19:36:53.649 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction test 2021-09-16T19:36:53.704 INFO:teuthology.orchestra.run.smithi109.stdout:Transaction test succeeded. 2021-09-16T19:36:53.708 INFO:teuthology.orchestra.run.smithi109.stdout:Running transaction 2021-09-16T19:36:53.710 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-nbd-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:36:53.830 INFO:teuthology.orchestra.run.smithi109.stdout: Preparing : 1/1 2021-09-16T19:36:53.939 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-nbd-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:36:54.042 INFO:teuthology.orchestra.run.smithi109.stdout: Installing : rbd-nbd-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:36:54.156 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-nbd-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:36:54.156 INFO:teuthology.orchestra.run.smithi118.stdout:Installed products updated. 2021-09-16T19:36:54.276 INFO:teuthology.orchestra.run.smithi109.stdout: Running scriptlet: rbd-nbd-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:36:54.519 INFO:teuthology.orchestra.run.smithi109.stdout: Verifying : rbd-nbd-2:17.0.0-7730.gbc749ef4.el8.x86_64 1/1 2021-09-16T19:36:54.520 INFO:teuthology.orchestra.run.smithi109.stdout:Installed products updated. 2021-09-16T19:37:04.616 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:36:31 PM UTC. 2021-09-16T19:37:04.786 INFO:teuthology.orchestra.run.smithi109.stdout:Last metadata expiration check: 0:00:05 ago on Thu 16 Sep 2021 07:36:31 PM UTC. 2021-09-16T19:37:07.115 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:07.116 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2021-09-16T19:37:07.116 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:37:07.116 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:07.116 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2021-09-16T19:37:07.246 DEBUG:teuthology.parallel:result is None 2021-09-16T19:37:07.458 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:07.458 INFO:teuthology.orchestra.run.smithi109.stdout:Installed: 2021-09-16T19:37:07.458 INFO:teuthology.orchestra.run.smithi109.stdout: rbd-nbd-2:17.0.0-7730.gbc749ef4.el8.x86_64 2021-09-16T19:37:07.459 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:07.459 INFO:teuthology.orchestra.run.smithi109.stdout:Complete! 2021-09-16T19:37:07.578 DEBUG:teuthology.parallel:result is None 2021-09-16T19:37:07.579 DEBUG:teuthology.orchestra.run.smithi109:> sudo lsb_release -is 2021-09-16T19:37:07.618 INFO:teuthology.orchestra.run.smithi109.stdout:RedHatEnterprise 2021-09-16T19:37:07.619 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-16T19:37:07.620 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=bc749ef43279601f3e9305d4ad8723d440abaa66 2021-09-16T19:37:07.774 DEBUG:teuthology.orchestra.run.smithi109:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-09-16T19:37:07.793 INFO:teuthology.orchestra.run.smithi109.stdout:17.0.0-7730.gbc749ef4.el8 2021-09-16T19:37:07.794 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-7730.gbc749ef4.el8 2021-09-16T19:37:07.795 INFO:teuthology.task.install:The correct ceph version 17.0.0-7730.gbc749ef4 is installed. 2021-09-16T19:37:07.795 DEBUG:teuthology.orchestra.run.smithi118:> sudo lsb_release -is 2021-09-16T19:37:07.836 INFO:teuthology.orchestra.run.smithi118.stdout:RedHatEnterprise 2021-09-16T19:37:07.837 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-16T19:37:07.838 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=bc749ef43279601f3e9305d4ad8723d440abaa66 2021-09-16T19:37:07.990 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-09-16T19:37:08.010 INFO:teuthology.orchestra.run.smithi118.stdout:17.0.0-7730.gbc749ef4.el8 2021-09-16T19:37:08.011 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-7730.gbc749ef4.el8 2021-09-16T19:37:08.011 INFO:teuthology.task.install:The correct ceph version 17.0.0-7730.gbc749ef4 is installed. 2021-09-16T19:37:08.012 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-09-16T19:37:08.012 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:08.013 DEBUG:teuthology.orchestra.run.smithi109:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-09-16T19:37:08.038 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:08.038 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-09-16T19:37:08.074 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-09-16T19:37:08.074 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:08.075 DEBUG:teuthology.orchestra.run.smithi109:> sudo dd of=/usr/bin/daemon-helper 2021-09-16T19:37:08.102 DEBUG:teuthology.orchestra.run.smithi109:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-09-16T19:37:08.165 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:08.166 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/daemon-helper 2021-09-16T19:37:08.191 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-09-16T19:37:08.254 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-09-16T19:37:08.255 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:08.255 DEBUG:teuthology.orchestra.run.smithi109:> sudo dd of=/usr/bin/adjust-ulimits 2021-09-16T19:37:08.279 DEBUG:teuthology.orchestra.run.smithi109:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-09-16T19:37:08.342 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:08.342 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/adjust-ulimits 2021-09-16T19:37:08.365 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-09-16T19:37:08.428 INFO:teuthology.run_tasks:Running task ceph... 2021-09-16T19:37:08.487 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-09-16T19:37:08.488 DEBUG:teuthology.orchestra.run.smithi109:> sudo chmod 777 /var/log/ceph 2021-09-16T19:37:08.491 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 777 /var/log/ceph 2021-09-16T19:37:08.514 INFO:tasks.ceph:Disabling ceph logrotate... 2021-09-16T19:37:08.514 DEBUG:teuthology.orchestra.run.smithi109:> sudo rm -f -- /etc/logrotate.d/ceph 2021-09-16T19:37:08.553 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/logrotate.d/ceph 2021-09-16T19:37:08.577 INFO:tasks.ceph:Creating extra log directories... 2021-09-16T19:37:08.577 DEBUG:teuthology.orchestra.run.smithi109:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-09-16T19:37:08.616 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-09-16T19:37:08.650 INFO:tasks.ceph:Creating ceph cluster c1... 2021-09-16T19:37:08.651 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-09-16T19:37:08.651 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mbenjamin-2021-09-16_15:12:17-rgw-wip-rgwadminops-fsid-distro-basic-smithi/6393866', 'branch': 'wip-rgwadminops-fsid', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '6393866', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'mbenjamin-2021-09-16_15:12:17-rgw-wip-rgwadminops-fsid-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-rgwadminops-fsid'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'bc749ef43279601f3e9305d4ad8723d440abaa66', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': 'bc749ef43279601f3e9305d4ad8723d440abaa66'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-rgw-sts-op-logs-plus-perl', 'sha1': '4e2e64ee99b6cbb34351964aad0f9df4874dea6d'}}, 'owner': 'scheduled_mbenjamin@shatner', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': 'bc749ef43279601f3e9305d4ad8723d440abaa66', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-rgw-sts-op-logs-plus-perl', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_4e2e64ee99b6cbb34351964aad0f9df4874dea6d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '4e2e64ee99b6cbb34351964aad0f9df4874dea6d', 'targets': {'smithi109.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDit7nKIpkndJnsa95XTFOPPxPMn5UoSxviofbE8fdv6otd42BKcm+tP55ory6N3+MkrJLaUuqFz++RRoIZKIWDcrH84eNqA4+6QouaQAgu+pE/Up1idQaX1XOmi+sPHX4sZTmcUJD1EAgS5otgTbEcMvcjdsA1HCJTSXjjBYyOFbiiw0GTgCn3ur6vmRzf55VaJX4KQYJc5ncAxgHHtVZmtUmKNRAbNqRWU0dALAwPbZPMXDoiL06z2mMoOQiEOPB9HeNfTuiumQhor88FAuB4QwswCJIUn7kWcdd+Q50B8C2xcULqP19KDmOPzZ6VQ+k5D2oMzY3hXf5NCrXZBb4OyOm8kiG/L2T4dva3ZA0TY/Gd7+dTeiob9BbjfRCf/aSc5QiZSSp5hcf2iUdcP1EeaDW6bNvp7A1u2QrP7truKxgtXs1J6w62p4JPwPuDGBWj6iN6wgVSmh7Cl6wnLGkgXJS3BA6DvvIQVZ4Mm/q5sTNPXajC8qAjF7gOaQXhO4s=', 'smithi118.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/mv+u+EjO7vwGqKRrKc1/4hE/xmnvsVd0Dz7wFkd88PDWJbemZ0yiW25IgARYTm1yKTAsbsE6T9o8nL9St/CSjrFYHTLfgL4e1XJ7JeNSOFEGMLye3FRvMZ+fBC3rYHO9i6zAP9PGrBtAxXfnx6TxfrPHuYM/uvHskbIccY0RgCC8ZtBtH5Ank0H+/eRfabQU0KlqamP1bvV7gDoSkhaGM7WJg+q9THZC0j3zvaN+q7fOQnjjvUPyClYttfIx8bYhaG7yGfxxuLzX+8bnQWJE4x5Hz2/aUl+9wgwvuK68Bt8mZebdgTj6gzWDZ9FEn5bfSTwXfP5rQmyqQt95CcEZHdS7KKFTP9tN8dwaJVxZDtbqqh/PsfuOo7L1ZDZQUINeGyV0qozcyHbM2O+GKMN4dGPCj2ByfmhAwn6ktXmQ/HOeNbZ8jTVpodUurhwKCd2OU2+dR1Y/Zu5m94V1g6ona4db93N+Z1l3Ry43ou9WOE7bwdhTGYNzeGznkFdq8lM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'bc749ef43279601f3e9305d4ad8723d440abaa66', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '63133ca92c299bd23f40f94d370cf5381b0b1313', 'timestamp': '2021-09-16_15:12:17', 'tube': 'smithi', 'user': 'mbenjamin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.30581'} 2021-09-16T19:37:08.652 DEBUG:teuthology.orchestra.run.smithi109:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-09-16T19:37:08.680 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-09-16T19:37:08.695 DEBUG:teuthology.orchestra.run.smithi109:> sudo install -d -m0777 -- /var/run/ceph 2021-09-16T19:37:08.734 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/run/ceph 2021-09-16T19:37:08.758 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:08.759 DEBUG:teuthology.orchestra.run.smithi109:> dd if=/scratch_devs of=/dev/stdout 2021-09-16T19:37:08.811 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-09-16T19:37:08.811 DEBUG:teuthology.orchestra.run.smithi109:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-09-16T19:37:08.865 INFO:teuthology.orchestra.run.smithi109.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-09-16T19:37:08.866 INFO:teuthology.orchestra.run.smithi109.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-16T19:37:08.866 INFO:teuthology.orchestra.run.smithi109.stdout:Device: 6h/6d Inode: 115778 Links: 1 2021-09-16T19:37:08.866 INFO:teuthology.orchestra.run.smithi109.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-16T19:37:08.866 INFO:teuthology.orchestra.run.smithi109.stdout:Context: system_u:object_r:device_t:s0 2021-09-16T19:37:08.867 INFO:teuthology.orchestra.run.smithi109.stdout:Access: 2021-09-16 19:36:26.302162461 +0000 2021-09-16T19:37:08.867 INFO:teuthology.orchestra.run.smithi109.stdout:Modify: 2021-09-16 19:27:25.821221023 +0000 2021-09-16T19:37:08.867 INFO:teuthology.orchestra.run.smithi109.stdout:Change: 2021-09-16 19:27:25.821221023 +0000 2021-09-16T19:37:08.867 INFO:teuthology.orchestra.run.smithi109.stdout: Birth: - 2021-09-16T19:37:08.874 INFO:teuthology.orchestra.run.smithi109.stderr:1+0 records in 2021-09-16T19:37:08.875 INFO:teuthology.orchestra.run.smithi109.stderr:1+0 records out 2021-09-16T19:37:08.875 INFO:teuthology.orchestra.run.smithi109.stderr:512 bytes copied, 0.000237371 s, 2.2 MB/s 2021-09-16T19:37:08.877 DEBUG:teuthology.orchestra.run.smithi109:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-09-16T19:37:08.932 INFO:teuthology.orchestra.run.smithi109.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-09-16T19:37:08.933 INFO:teuthology.orchestra.run.smithi109.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-16T19:37:08.933 INFO:teuthology.orchestra.run.smithi109.stdout:Device: 6h/6d Inode: 108387 Links: 1 2021-09-16T19:37:08.933 INFO:teuthology.orchestra.run.smithi109.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-16T19:37:08.933 INFO:teuthology.orchestra.run.smithi109.stdout:Context: system_u:object_r:device_t:s0 2021-09-16T19:37:08.933 INFO:teuthology.orchestra.run.smithi109.stdout:Access: 2021-09-16 19:36:26.302162461 +0000 2021-09-16T19:37:08.934 INFO:teuthology.orchestra.run.smithi109.stdout:Modify: 2021-09-16 19:27:26.172213400 +0000 2021-09-16T19:37:08.934 INFO:teuthology.orchestra.run.smithi109.stdout:Change: 2021-09-16 19:27:26.172213400 +0000 2021-09-16T19:37:08.934 INFO:teuthology.orchestra.run.smithi109.stdout: Birth: - 2021-09-16T19:37:08.941 INFO:teuthology.orchestra.run.smithi109.stderr:1+0 records in 2021-09-16T19:37:08.942 INFO:teuthology.orchestra.run.smithi109.stderr:1+0 records out 2021-09-16T19:37:08.942 INFO:teuthology.orchestra.run.smithi109.stderr:512 bytes copied, 0.000212076 s, 2.4 MB/s 2021-09-16T19:37:08.944 DEBUG:teuthology.orchestra.run.smithi109:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-09-16T19:37:08.998 INFO:teuthology.orchestra.run.smithi109.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-09-16T19:37:08.998 INFO:teuthology.orchestra.run.smithi109.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-16T19:37:08.999 INFO:teuthology.orchestra.run.smithi109.stdout:Device: 6h/6d Inode: 112341 Links: 1 2021-09-16T19:37:08.999 INFO:teuthology.orchestra.run.smithi109.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-16T19:37:08.999 INFO:teuthology.orchestra.run.smithi109.stdout:Context: system_u:object_r:device_t:s0 2021-09-16T19:37:08.999 INFO:teuthology.orchestra.run.smithi109.stdout:Access: 2021-09-16 19:36:26.302162461 +0000 2021-09-16T19:37:08.999 INFO:teuthology.orchestra.run.smithi109.stdout:Modify: 2021-09-16 19:27:26.518205885 +0000 2021-09-16T19:37:09.000 INFO:teuthology.orchestra.run.smithi109.stdout:Change: 2021-09-16 19:27:26.518205885 +0000 2021-09-16T19:37:09.000 INFO:teuthology.orchestra.run.smithi109.stdout: Birth: - 2021-09-16T19:37:09.009 INFO:teuthology.orchestra.run.smithi109.stderr:1+0 records in 2021-09-16T19:37:09.009 INFO:teuthology.orchestra.run.smithi109.stderr:1+0 records out 2021-09-16T19:37:09.009 INFO:teuthology.orchestra.run.smithi109.stderr:512 bytes copied, 0.000224413 s, 2.3 MB/s 2021-09-16T19:37:09.010 DEBUG:teuthology.orchestra.run.smithi109:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-09-16T19:37:09.065 INFO:teuthology.orchestra.run.smithi109.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-09-16T19:37:09.065 INFO:teuthology.orchestra.run.smithi109.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-16T19:37:09.066 INFO:teuthology.orchestra.run.smithi109.stdout:Device: 6h/6d Inode: 108516 Links: 1 2021-09-16T19:37:09.066 INFO:teuthology.orchestra.run.smithi109.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-16T19:37:09.066 INFO:teuthology.orchestra.run.smithi109.stdout:Context: system_u:object_r:device_t:s0 2021-09-16T19:37:09.066 INFO:teuthology.orchestra.run.smithi109.stdout:Access: 2021-09-16 19:36:26.303162440 +0000 2021-09-16T19:37:09.066 INFO:teuthology.orchestra.run.smithi109.stdout:Modify: 2021-09-16 19:27:26.856198544 +0000 2021-09-16T19:37:09.067 INFO:teuthology.orchestra.run.smithi109.stdout:Change: 2021-09-16 19:27:26.856198544 +0000 2021-09-16T19:37:09.067 INFO:teuthology.orchestra.run.smithi109.stdout: Birth: - 2021-09-16T19:37:09.074 INFO:teuthology.orchestra.run.smithi109.stderr:1+0 records in 2021-09-16T19:37:09.075 INFO:teuthology.orchestra.run.smithi109.stderr:1+0 records out 2021-09-16T19:37:09.075 INFO:teuthology.orchestra.run.smithi109.stderr:512 bytes copied, 0.000181812 s, 2.8 MB/s 2021-09-16T19:37:09.077 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-16T19:37:09.078 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi109.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-16T19:37:09.078 INFO:tasks.ceph:Generating config... 2021-09-16T19:37:09.080 INFO:tasks.ceph:[client] debug rgw = 20 2021-09-16T19:37:09.080 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-09-16T19:37:09.080 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-09-16T19:37:09.081 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-16T19:37:09.081 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-09-16T19:37:09.081 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-09-16T19:37:09.081 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-09-16T19:37:09.082 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-09-16T19:37:09.082 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2021-09-16T19:37:09.082 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-09-16T19:37:09.083 INFO:tasks.ceph:[client] setgroup = ceph 2021-09-16T19:37:09.083 INFO:tasks.ceph:[client] setuser = ceph 2021-09-16T19:37:09.083 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-09-16T19:37:09.084 INFO:tasks.ceph:[mgr] debug ms = 1 2021-09-16T19:37:09.084 INFO:tasks.ceph:[mon] debug mon = 20 2021-09-16T19:37:09.084 INFO:tasks.ceph:[mon] debug ms = 1 2021-09-16T19:37:09.085 INFO:tasks.ceph:[mon] debug paxos = 20 2021-09-16T19:37:09.085 INFO:tasks.ceph:[osd] debug ms = 1 2021-09-16T19:37:09.085 INFO:tasks.ceph:[osd] debug osd = 20 2021-09-16T19:37:09.086 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-09-16T19:37:09.086 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-09-16T19:37:09.086 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-09-16T19:37:09.086 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-09-16T19:37:09.087 INFO:tasks.ceph:Setting up c1.mon.a... 2021-09-16T19:37:09.087 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2021-09-16T19:37:09.167 INFO:teuthology.orchestra.run.smithi109.stdout:creating /etc/ceph/c1.keyring 2021-09-16T19:37:09.169 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2021-09-16T19:37:09.208 DEBUG:teuthology.orchestra.run.smithi109:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-16T19:37:09.272 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.109')] 2021-09-16T19:37:09.273 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.109'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-09-16T19:37:09.274 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:09.274 DEBUG:teuthology.orchestra.run.smithi109:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-09-16T19:37:09.326 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.109 --print /home/ubuntu/cephtest/c1.monmap 2021-09-16T19:37:09.401 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:37:09.401 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:37:09.402 INFO:teuthology.orchestra.run.smithi109.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2021-09-16T19:37:09.403 INFO:teuthology.orchestra.run.smithi109.stdout:monmaptool: generated fsid 3c353c67-c60a-4c36-92c7-ea526a847dac 2021-09-16T19:37:09.403 INFO:teuthology.orchestra.run.smithi109.stdout:setting min_mon_release = octopus 2021-09-16T19:37:09.403 INFO:teuthology.orchestra.run.smithi109.stdout:epoch 0 2021-09-16T19:37:09.404 INFO:teuthology.orchestra.run.smithi109.stdout:fsid 3c353c67-c60a-4c36-92c7-ea526a847dac 2021-09-16T19:37:09.404 INFO:teuthology.orchestra.run.smithi109.stdout:last_changed 2021-09-16T19:37:09.401287+0000 2021-09-16T19:37:09.404 INFO:teuthology.orchestra.run.smithi109.stdout:created 2021-09-16T19:37:09.401287+0000 2021-09-16T19:37:09.404 INFO:teuthology.orchestra.run.smithi109.stdout:min_mon_release 15 (octopus) 2021-09-16T19:37:09.404 INFO:teuthology.orchestra.run.smithi109.stdout:election_strategy: 1 2021-09-16T19:37:09.405 INFO:teuthology.orchestra.run.smithi109.stdout:0: [v2:172.21.15.109:3300/0,v1:172.21.15.109:6789/0] mon.a 2021-09-16T19:37:09.405 INFO:teuthology.orchestra.run.smithi109.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2021-09-16T19:37:09.406 DEBUG:teuthology.orchestra.run.smithi109:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-09-16T19:37:09.457 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 3c353c67-c60a-4c36-92c7-ea526a847dac... 2021-09-16T19:37:09.458 DEBUG:teuthology.orchestra.run.smithi109:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-09-16T19:37:09.499 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-09-16T19:37:09.539 INFO:teuthology.orchestra.run.smithi109.stdout:[global] 2021-09-16T19:37:09.539 INFO:teuthology.orchestra.run.smithi109.stdout: chdir = "" 2021-09-16T19:37:09.539 INFO:teuthology.orchestra.run.smithi109.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-16T19:37:09.540 INFO:teuthology.orchestra.run.smithi109.stdout: auth supported = cephx 2021-09-16T19:37:09.540 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.540 INFO:teuthology.orchestra.run.smithi109.stdout: filestore xattr use omap = true 2021-09-16T19:37:09.541 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.541 INFO:teuthology.orchestra.run.smithi109.stdout: mon clock drift allowed = 1.000 2021-09-16T19:37:09.541 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.542 INFO:teuthology.orchestra.run.smithi109.stdout: osd crush chooseleaf type = 0 2021-09-16T19:37:09.542 INFO:teuthology.orchestra.run.smithi109.stdout: auth debug = true 2021-09-16T19:37:09.542 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.542 INFO:teuthology.orchestra.run.smithi109.stdout: ms die on old message = true 2021-09-16T19:37:09.543 INFO:teuthology.orchestra.run.smithi109.stdout: ms die on bug = true 2021-09-16T19:37:09.543 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.543 INFO:teuthology.orchestra.run.smithi109.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-16T19:37:09.543 INFO:teuthology.orchestra.run.smithi109.stdout: mon pg warn max object skew = 0 2021-09-16T19:37:09.543 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.544 INFO:teuthology.orchestra.run.smithi109.stdout: # disable pg_autoscaler by default for new pools 2021-09-16T19:37:09.544 INFO:teuthology.orchestra.run.smithi109.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-16T19:37:09.544 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.544 INFO:teuthology.orchestra.run.smithi109.stdout: osd pool default size = 2 2021-09-16T19:37:09.545 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.545 INFO:teuthology.orchestra.run.smithi109.stdout: mon osd allow primary affinity = true 2021-09-16T19:37:09.545 INFO:teuthology.orchestra.run.smithi109.stdout: mon osd allow pg remap = true 2021-09-16T19:37:09.545 INFO:teuthology.orchestra.run.smithi109.stdout: mon warn on legacy crush tunables = false 2021-09-16T19:37:09.545 INFO:teuthology.orchestra.run.smithi109.stdout: mon warn on crush straw calc version zero = false 2021-09-16T19:37:09.546 INFO:teuthology.orchestra.run.smithi109.stdout: mon warn on no sortbitwise = false 2021-09-16T19:37:09.546 INFO:teuthology.orchestra.run.smithi109.stdout: mon warn on osd down out interval zero = false 2021-09-16T19:37:09.546 INFO:teuthology.orchestra.run.smithi109.stdout: mon warn on too few osds = false 2021-09-16T19:37:09.546 INFO:teuthology.orchestra.run.smithi109.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-16T19:37:09.546 INFO:teuthology.orchestra.run.smithi109.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-16T19:37:09.547 INFO:teuthology.orchestra.run.smithi109.stdout: mon_allow_pool_size_one = true 2021-09-16T19:37:09.547 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.547 INFO:teuthology.orchestra.run.smithi109.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-16T19:37:09.547 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.548 INFO:teuthology.orchestra.run.smithi109.stdout: osd default data pool replay window = 5 2021-09-16T19:37:09.548 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.548 INFO:teuthology.orchestra.run.smithi109.stdout: mon allow pool delete = true 2021-09-16T19:37:09.548 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.548 INFO:teuthology.orchestra.run.smithi109.stdout: mon cluster log file level = debug 2021-09-16T19:37:09.549 INFO:teuthology.orchestra.run.smithi109.stdout: debug asserts on shutdown = true 2021-09-16T19:37:09.549 INFO:teuthology.orchestra.run.smithi109.stdout: mon health detail to clog = false 2021-09-16T19:37:09.549 INFO:teuthology.orchestra.run.smithi109.stdout: mon host = 172.21.15.109 2021-09-16T19:37:09.549 INFO:teuthology.orchestra.run.smithi109.stdout: fsid = 3c353c67-c60a-4c36-92c7-ea526a847dac 2021-09-16T19:37:09.550 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.550 INFO:teuthology.orchestra.run.smithi109.stdout:[osd] 2021-09-16T19:37:09.550 INFO:teuthology.orchestra.run.smithi109.stdout: osd journal size = 100 2021-09-16T19:37:09.550 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.550 INFO:teuthology.orchestra.run.smithi109.stdout: osd scrub load threshold = 5.0 2021-09-16T19:37:09.551 INFO:teuthology.orchestra.run.smithi109.stdout: osd scrub max interval = 600 2021-09-16T19:37:09.551 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.551 INFO:teuthology.orchestra.run.smithi109.stdout: osd recover clone overlap = true 2021-09-16T19:37:09.551 INFO:teuthology.orchestra.run.smithi109.stdout: osd recovery max chunk = 1048576 2021-09-16T19:37:09.552 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.552 INFO:teuthology.orchestra.run.smithi109.stdout: osd debug shutdown = true 2021-09-16T19:37:09.552 INFO:teuthology.orchestra.run.smithi109.stdout: osd debug op order = true 2021-09-16T19:37:09.552 INFO:teuthology.orchestra.run.smithi109.stdout: osd debug verify stray on activate = true 2021-09-16T19:37:09.553 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.553 INFO:teuthology.orchestra.run.smithi109.stdout: osd open classes on start = true 2021-09-16T19:37:09.553 INFO:teuthology.orchestra.run.smithi109.stdout: osd debug pg log writeout = true 2021-09-16T19:37:09.554 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.554 INFO:teuthology.orchestra.run.smithi109.stdout: osd deep scrub update digest min age = 30 2021-09-16T19:37:09.554 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.554 INFO:teuthology.orchestra.run.smithi109.stdout: osd map max advance = 10 2021-09-16T19:37:09.555 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.555 INFO:teuthology.orchestra.run.smithi109.stdout: journal zero on create = true 2021-09-16T19:37:09.555 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.555 INFO:teuthology.orchestra.run.smithi109.stdout: filestore ondisk finisher threads = 3 2021-09-16T19:37:09.555 INFO:teuthology.orchestra.run.smithi109.stdout: filestore apply finisher threads = 3 2021-09-16T19:37:09.556 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.556 INFO:teuthology.orchestra.run.smithi109.stdout: bdev debug aio = true 2021-09-16T19:37:09.556 INFO:teuthology.orchestra.run.smithi109.stdout: osd debug misdirected ops = true 2021-09-16T19:37:09.556 INFO:teuthology.orchestra.run.smithi109.stdout: debug ms = 1 2021-09-16T19:37:09.557 INFO:teuthology.orchestra.run.smithi109.stdout: debug osd = 20 2021-09-16T19:37:09.557 INFO:teuthology.orchestra.run.smithi109.stdout: rocksdb delete range threshold = 0 2021-09-16T19:37:09.557 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.557 INFO:teuthology.orchestra.run.smithi109.stdout:[mgr] 2021-09-16T19:37:09.557 INFO:teuthology.orchestra.run.smithi109.stdout: debug ms = 1 2021-09-16T19:37:09.558 INFO:teuthology.orchestra.run.smithi109.stdout: debug mgr = 20 2021-09-16T19:37:09.558 INFO:teuthology.orchestra.run.smithi109.stdout: debug mon = 20 2021-09-16T19:37:09.558 INFO:teuthology.orchestra.run.smithi109.stdout: debug auth = 20 2021-09-16T19:37:09.558 INFO:teuthology.orchestra.run.smithi109.stdout: mon reweight min pgs per osd = 4 2021-09-16T19:37:09.559 INFO:teuthology.orchestra.run.smithi109.stdout: mon reweight min bytes per osd = 10 2021-09-16T19:37:09.559 INFO:teuthology.orchestra.run.smithi109.stdout: mgr/telemetry/nag = false 2021-09-16T19:37:09.559 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.559 INFO:teuthology.orchestra.run.smithi109.stdout:[mon] 2021-09-16T19:37:09.559 INFO:teuthology.orchestra.run.smithi109.stdout: debug ms = 1 2021-09-16T19:37:09.560 INFO:teuthology.orchestra.run.smithi109.stdout: debug mon = 20 2021-09-16T19:37:09.560 INFO:teuthology.orchestra.run.smithi109.stdout: debug paxos = 20 2021-09-16T19:37:09.560 INFO:teuthology.orchestra.run.smithi109.stdout: debug auth = 20 2021-09-16T19:37:09.560 INFO:teuthology.orchestra.run.smithi109.stdout: mon data avail warn = 5 2021-09-16T19:37:09.561 INFO:teuthology.orchestra.run.smithi109.stdout: mon mgr mkfs grace = 240 2021-09-16T19:37:09.561 INFO:teuthology.orchestra.run.smithi109.stdout: mon reweight min pgs per osd = 4 2021-09-16T19:37:09.561 INFO:teuthology.orchestra.run.smithi109.stdout: mon osd reporter subtree level = osd 2021-09-16T19:37:09.562 INFO:teuthology.orchestra.run.smithi109.stdout: mon osd prime pg temp = true 2021-09-16T19:37:09.563 INFO:teuthology.orchestra.run.smithi109.stdout: mon reweight min bytes per osd = 10 2021-09-16T19:37:09.563 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.563 INFO:teuthology.orchestra.run.smithi109.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-16T19:37:09.564 INFO:teuthology.orchestra.run.smithi109.stdout: auth mon ticket ttl = 660 # 11m 2021-09-16T19:37:09.564 INFO:teuthology.orchestra.run.smithi109.stdout: auth service ticket ttl = 240 # 4m 2021-09-16T19:37:09.564 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.565 INFO:teuthology.orchestra.run.smithi109.stdout: # don't complain about insecure global_id in the test suite 2021-09-16T19:37:09.565 INFO:teuthology.orchestra.run.smithi109.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-16T19:37:09.565 INFO:teuthology.orchestra.run.smithi109.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-16T19:37:09.565 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:09.566 INFO:teuthology.orchestra.run.smithi109.stdout:[client] 2021-09-16T19:37:09.566 INFO:teuthology.orchestra.run.smithi109.stdout: rgw cache enabled = true 2021-09-16T19:37:09.566 INFO:teuthology.orchestra.run.smithi109.stdout: rgw enable ops log = true 2021-09-16T19:37:09.566 INFO:teuthology.orchestra.run.smithi109.stdout: rgw enable usage log = true 2021-09-16T19:37:09.566 INFO:teuthology.orchestra.run.smithi109.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-16T19:37:09.567 INFO:teuthology.orchestra.run.smithi109.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-16T19:37:09.567 INFO:teuthology.orchestra.run.smithi109.stdout: debug rgw = 20 2021-09-16T19:37:09.567 INFO:teuthology.orchestra.run.smithi109.stdout: rgw crypt require ssl = False 2021-09-16T19:37:09.567 INFO:teuthology.orchestra.run.smithi109.stdout: rgw crypt s3 kms backend = testing 2021-09-16T19:37:09.568 INFO:teuthology.orchestra.run.smithi109.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-16T19:37:09.568 INFO:teuthology.orchestra.run.smithi109.stdout: rgw curl low speed time = 300 2021-09-16T19:37:09.568 INFO:teuthology.orchestra.run.smithi109.stdout: rgw data log num shards = 4 2021-09-16T19:37:09.568 INFO:teuthology.orchestra.run.smithi109.stdout: rgw md log max shards = 4 2021-09-16T19:37:09.568 INFO:teuthology.orchestra.run.smithi109.stdout: rgw sync log trim interval = 0 2021-09-16T19:37:09.569 INFO:teuthology.orchestra.run.smithi109.stdout: rgw sync meta inject err probability = 0.1 2021-09-16T19:37:09.569 INFO:teuthology.orchestra.run.smithi109.stdout: rgw sync obj etag verify = True 2021-09-16T19:37:09.569 INFO:teuthology.orchestra.run.smithi109.stdout: setgroup = ceph 2021-09-16T19:37:09.569 INFO:teuthology.orchestra.run.smithi109.stdout: setuser = ceph 2021-09-16T19:37:09.570 INFO:teuthology.orchestra.run.smithi109.stdout:[mon.a] 2021-09-16T19:37:09.570 INFO:teuthology.orchestra.run.smithi109.stdout:[osd.0] 2021-09-16T19:37:09.570 INFO:teuthology.orchestra.run.smithi109.stdout: osd_max_omap_entries_per_request = 10 2021-09-16T19:37:09.570 INFO:teuthology.orchestra.run.smithi109.stdout:[osd.1] 2021-09-16T19:37:09.571 INFO:teuthology.orchestra.run.smithi109.stdout: osd_max_omap_entries_per_request = 1000 2021-09-16T19:37:09.571 INFO:teuthology.orchestra.run.smithi109.stdout:[osd.2] 2021-09-16T19:37:09.571 INFO:teuthology.orchestra.run.smithi109.stdout: osd_max_omap_entries_per_request = 10000 2021-09-16T19:37:09.573 INFO:teuthology.orchestra.run.smithi118.stdout:[global] 2021-09-16T19:37:09.573 INFO:teuthology.orchestra.run.smithi118.stdout: chdir = "" 2021-09-16T19:37:09.573 INFO:teuthology.orchestra.run.smithi118.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-16T19:37:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: auth supported = cephx 2021-09-16T19:37:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: filestore xattr use omap = true 2021-09-16T19:37:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: mon clock drift allowed = 1.000 2021-09-16T19:37:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: osd crush chooseleaf type = 0 2021-09-16T19:37:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: auth debug = true 2021-09-16T19:37:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on old message = true 2021-09-16T19:37:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on bug = true 2021-09-16T19:37:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-16T19:37:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: mon pg warn max object skew = 0 2021-09-16T19:37:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: # disable pg_autoscaler by default for new pools 2021-09-16T19:37:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-16T19:37:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default size = 2 2021-09-16T19:37:09.578 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.578 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow primary affinity = true 2021-09-16T19:37:09.578 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow pg remap = true 2021-09-16T19:37:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on legacy crush tunables = false 2021-09-16T19:37:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on crush straw calc version zero = false 2021-09-16T19:37:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on no sortbitwise = false 2021-09-16T19:37:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on osd down out interval zero = false 2021-09-16T19:37:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on too few osds = false 2021-09-16T19:37:09.580 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-16T19:37:09.580 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-16T19:37:09.580 INFO:teuthology.orchestra.run.smithi118.stdout: mon_allow_pool_size_one = true 2021-09-16T19:37:09.580 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-16T19:37:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: osd default data pool replay window = 5 2021-09-16T19:37:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: mon allow pool delete = true 2021-09-16T19:37:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: mon cluster log file level = debug 2021-09-16T19:37:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: debug asserts on shutdown = true 2021-09-16T19:37:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: mon health detail to clog = false 2021-09-16T19:37:09.583 INFO:teuthology.orchestra.run.smithi118.stdout: mon host = 172.21.15.109 2021-09-16T19:37:09.583 INFO:teuthology.orchestra.run.smithi118.stdout: fsid = 3c353c67-c60a-4c36-92c7-ea526a847dac 2021-09-16T19:37:09.583 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.583 INFO:teuthology.orchestra.run.smithi118.stdout:[osd] 2021-09-16T19:37:09.584 INFO:teuthology.orchestra.run.smithi118.stdout: osd journal size = 100 2021-09-16T19:37:09.584 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.584 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub load threshold = 5.0 2021-09-16T19:37:09.584 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub max interval = 600 2021-09-16T19:37:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: osd recover clone overlap = true 2021-09-16T19:37:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: osd recovery max chunk = 1048576 2021-09-16T19:37:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug shutdown = true 2021-09-16T19:37:09.586 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug op order = true 2021-09-16T19:37:09.586 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify stray on activate = true 2021-09-16T19:37:09.586 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.586 INFO:teuthology.orchestra.run.smithi118.stdout: osd open classes on start = true 2021-09-16T19:37:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug pg log writeout = true 2021-09-16T19:37:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: osd deep scrub update digest min age = 30 2021-09-16T19:37:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.588 INFO:teuthology.orchestra.run.smithi118.stdout: osd map max advance = 10 2021-09-16T19:37:09.588 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.588 INFO:teuthology.orchestra.run.smithi118.stdout: journal zero on create = true 2021-09-16T19:37:09.588 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.588 INFO:teuthology.orchestra.run.smithi118.stdout: filestore ondisk finisher threads = 3 2021-09-16T19:37:09.589 INFO:teuthology.orchestra.run.smithi118.stdout: filestore apply finisher threads = 3 2021-09-16T19:37:09.589 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.589 INFO:teuthology.orchestra.run.smithi118.stdout: bdev debug aio = true 2021-09-16T19:37:09.589 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug misdirected ops = true 2021-09-16T19:37:09.589 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2021-09-16T19:37:09.590 INFO:teuthology.orchestra.run.smithi118.stdout: debug osd = 20 2021-09-16T19:37:09.590 INFO:teuthology.orchestra.run.smithi118.stdout: rocksdb delete range threshold = 0 2021-09-16T19:37:09.590 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.590 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr] 2021-09-16T19:37:09.591 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2021-09-16T19:37:09.591 INFO:teuthology.orchestra.run.smithi118.stdout: debug mgr = 20 2021-09-16T19:37:09.591 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2021-09-16T19:37:09.591 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2021-09-16T19:37:09.592 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2021-09-16T19:37:09.592 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2021-09-16T19:37:09.592 INFO:teuthology.orchestra.run.smithi118.stdout: mgr/telemetry/nag = false 2021-09-16T19:37:09.592 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.592 INFO:teuthology.orchestra.run.smithi118.stdout:[mon] 2021-09-16T19:37:09.593 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2021-09-16T19:37:09.593 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2021-09-16T19:37:09.593 INFO:teuthology.orchestra.run.smithi118.stdout: debug paxos = 20 2021-09-16T19:37:09.593 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2021-09-16T19:37:09.593 INFO:teuthology.orchestra.run.smithi118.stdout: mon data avail warn = 5 2021-09-16T19:37:09.594 INFO:teuthology.orchestra.run.smithi118.stdout: mon mgr mkfs grace = 240 2021-09-16T19:37:09.594 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2021-09-16T19:37:09.594 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd reporter subtree level = osd 2021-09-16T19:37:09.594 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd prime pg temp = true 2021-09-16T19:37:09.595 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2021-09-16T19:37:09.595 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.595 INFO:teuthology.orchestra.run.smithi118.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-16T19:37:09.595 INFO:teuthology.orchestra.run.smithi118.stdout: auth mon ticket ttl = 660 # 11m 2021-09-16T19:37:09.595 INFO:teuthology.orchestra.run.smithi118.stdout: auth service ticket ttl = 240 # 4m 2021-09-16T19:37:09.596 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.596 INFO:teuthology.orchestra.run.smithi118.stdout: # don't complain about insecure global_id in the test suite 2021-09-16T19:37:09.596 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-16T19:37:09.596 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-16T19:37:09.597 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:09.597 INFO:teuthology.orchestra.run.smithi118.stdout:[client] 2021-09-16T19:37:09.597 INFO:teuthology.orchestra.run.smithi118.stdout: rgw cache enabled = true 2021-09-16T19:37:09.597 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable ops log = true 2021-09-16T19:37:09.597 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable usage log = true 2021-09-16T19:37:09.598 INFO:teuthology.orchestra.run.smithi118.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-16T19:37:09.598 INFO:teuthology.orchestra.run.smithi118.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-16T19:37:09.598 INFO:teuthology.orchestra.run.smithi118.stdout: debug rgw = 20 2021-09-16T19:37:09.598 INFO:teuthology.orchestra.run.smithi118.stdout: rgw crypt require ssl = False 2021-09-16T19:37:09.599 INFO:teuthology.orchestra.run.smithi118.stdout: rgw crypt s3 kms backend = testing 2021-09-16T19:37:09.599 INFO:teuthology.orchestra.run.smithi118.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-16T19:37:09.599 INFO:teuthology.orchestra.run.smithi118.stdout: rgw curl low speed time = 300 2021-09-16T19:37:09.599 INFO:teuthology.orchestra.run.smithi118.stdout: rgw data log num shards = 4 2021-09-16T19:37:09.599 INFO:teuthology.orchestra.run.smithi118.stdout: rgw md log max shards = 4 2021-09-16T19:37:09.600 INFO:teuthology.orchestra.run.smithi118.stdout: rgw sync log trim interval = 0 2021-09-16T19:37:09.600 INFO:teuthology.orchestra.run.smithi118.stdout: rgw sync meta inject err probability = 0.1 2021-09-16T19:37:09.600 INFO:teuthology.orchestra.run.smithi118.stdout: rgw sync obj etag verify = True 2021-09-16T19:37:09.600 INFO:teuthology.orchestra.run.smithi118.stdout: setgroup = ceph 2021-09-16T19:37:09.601 INFO:teuthology.orchestra.run.smithi118.stdout: setuser = ceph 2021-09-16T19:37:09.601 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.a] 2021-09-16T19:37:09.601 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.0] 2021-09-16T19:37:09.601 INFO:teuthology.orchestra.run.smithi118.stdout: osd_max_omap_entries_per_request = 10 2021-09-16T19:37:09.601 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.1] 2021-09-16T19:37:09.602 INFO:teuthology.orchestra.run.smithi118.stdout: osd_max_omap_entries_per_request = 1000 2021-09-16T19:37:09.602 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.2] 2021-09-16T19:37:09.602 INFO:teuthology.orchestra.run.smithi118.stdout: osd_max_omap_entries_per_request = 10000 2021-09-16T19:37:09.603 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2021-09-16T19:37:09.603 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2021-09-16T19:37:09.661 INFO:tasks.ceph:Copying monmap to all nodes... 2021-09-16T19:37:09.662 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:09.662 DEBUG:teuthology.orchestra.run.smithi109:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2021-09-16T19:37:09.677 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:09.677 DEBUG:teuthology.orchestra.run.smithi109:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2021-09-16T19:37:09.731 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi109.front.sepia.ceph.com 2021-09-16T19:37:09.731 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:09.732 DEBUG:teuthology.orchestra.run.smithi109:> sudo dd of=/etc/ceph/c1.keyring 2021-09-16T19:37:09.732 DEBUG:teuthology.orchestra.run.smithi109:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-16T19:37:09.803 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:09.804 DEBUG:teuthology.orchestra.run.smithi109:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-09-16T19:37:09.858 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi118.front.sepia.ceph.com 2021-09-16T19:37:09.858 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:09.859 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/c1.keyring 2021-09-16T19:37:09.859 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-16T19:37:09.895 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:09.896 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-09-16T19:37:09.950 INFO:tasks.ceph:Setting up mon nodes... 2021-09-16T19:37:09.951 INFO:tasks.ceph:Setting up mgr nodes... 2021-09-16T19:37:09.951 DEBUG:teuthology.orchestra.run.smithi109:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2021-09-16T19:37:09.997 INFO:teuthology.orchestra.run.smithi109.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2021-09-16T19:37:09.999 INFO:tasks.ceph:Setting up mds nodes... 2021-09-16T19:37:10.000 INFO:tasks.ceph_client:Setting up client nodes... 2021-09-16T19:37:10.001 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2021-09-16T19:37:10.038 INFO:teuthology.orchestra.run.smithi109.stdout:creating /etc/ceph/c1.client.0.keyring 2021-09-16T19:37:10.050 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2021-09-16T19:37:10.126 INFO:teuthology.orchestra.run.smithi109.stdout:creating /etc/ceph/c1.client.1.keyring 2021-09-16T19:37:10.139 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-09-16T19:37:10.139 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi109.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-16T19:37:10.139 DEBUG:teuthology.orchestra.run.smithi109:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2021-09-16T19:37:10.202 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-16T19:37:10.203 INFO:tasks.ceph:role: c1.osd.0 2021-09-16T19:37:10.203 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi109.front.sepia.ceph.com 2021-09-16T19:37:10.203 DEBUG:teuthology.orchestra.run.smithi109:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-09-16T19:37:10.273 INFO:teuthology.orchestra.run.smithi109.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-09-16T19:37:10.273 INFO:teuthology.orchestra.run.smithi109.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-16T19:37:10.273 INFO:teuthology.orchestra.run.smithi109.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-16T19:37:10.273 INFO:teuthology.orchestra.run.smithi109.stdout: = reflink=1 2021-09-16T19:37:10.274 INFO:teuthology.orchestra.run.smithi109.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-16T19:37:10.274 INFO:teuthology.orchestra.run.smithi109.stdout: = sunit=0 swidth=0 blks 2021-09-16T19:37:10.274 INFO:teuthology.orchestra.run.smithi109.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-16T19:37:10.274 INFO:teuthology.orchestra.run.smithi109.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-16T19:37:10.274 INFO:teuthology.orchestra.run.smithi109.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-16T19:37:10.275 INFO:teuthology.orchestra.run.smithi109.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-16T19:37:10.639 INFO:teuthology.orchestra.run.smithi109.stdout:Discarding blocks...Done. 2021-09-16T19:37:10.680 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi109.front.sepia.ceph.com -o noatime 2021-09-16T19:37:10.680 DEBUG:teuthology.orchestra.run.smithi109:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2021-09-16T19:37:10.802 DEBUG:teuthology.orchestra.run.smithi109:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2021-09-16T19:37:10.830 DEBUG:teuthology.orchestra.run.smithi109:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2021-09-16T19:37:10.894 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-16T19:37:10.894 INFO:tasks.ceph:role: c1.osd.1 2021-09-16T19:37:10.895 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi109.front.sepia.ceph.com 2021-09-16T19:37:10.895 DEBUG:teuthology.orchestra.run.smithi109:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-09-16T19:37:10.965 INFO:teuthology.orchestra.run.smithi109.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-09-16T19:37:10.965 INFO:teuthology.orchestra.run.smithi109.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-16T19:37:10.965 INFO:teuthology.orchestra.run.smithi109.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-16T19:37:10.966 INFO:teuthology.orchestra.run.smithi109.stdout: = reflink=1 2021-09-16T19:37:10.966 INFO:teuthology.orchestra.run.smithi109.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-16T19:37:10.966 INFO:teuthology.orchestra.run.smithi109.stdout: = sunit=0 swidth=0 blks 2021-09-16T19:37:10.966 INFO:teuthology.orchestra.run.smithi109.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-16T19:37:10.967 INFO:teuthology.orchestra.run.smithi109.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-16T19:37:10.967 INFO:teuthology.orchestra.run.smithi109.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-16T19:37:10.967 INFO:teuthology.orchestra.run.smithi109.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-16T19:37:11.324 INFO:teuthology.orchestra.run.smithi109.stdout:Discarding blocks...Done. 2021-09-16T19:37:11.326 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi109.front.sepia.ceph.com -o noatime 2021-09-16T19:37:11.327 DEBUG:teuthology.orchestra.run.smithi109:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2021-09-16T19:37:11.412 DEBUG:teuthology.orchestra.run.smithi109:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2021-09-16T19:37:11.439 DEBUG:teuthology.orchestra.run.smithi109:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2021-09-16T19:37:11.503 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-16T19:37:11.503 INFO:tasks.ceph:role: c1.osd.2 2021-09-16T19:37:11.503 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi109.front.sepia.ceph.com 2021-09-16T19:37:11.504 DEBUG:teuthology.orchestra.run.smithi109:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-09-16T19:37:11.574 INFO:teuthology.orchestra.run.smithi109.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-09-16T19:37:11.574 INFO:teuthology.orchestra.run.smithi109.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-16T19:37:11.575 INFO:teuthology.orchestra.run.smithi109.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-16T19:37:11.575 INFO:teuthology.orchestra.run.smithi109.stdout: = reflink=1 2021-09-16T19:37:11.575 INFO:teuthology.orchestra.run.smithi109.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-16T19:37:11.575 INFO:teuthology.orchestra.run.smithi109.stdout: = sunit=0 swidth=0 blks 2021-09-16T19:37:11.576 INFO:teuthology.orchestra.run.smithi109.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-16T19:37:11.576 INFO:teuthology.orchestra.run.smithi109.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-16T19:37:11.576 INFO:teuthology.orchestra.run.smithi109.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-16T19:37:11.577 INFO:teuthology.orchestra.run.smithi109.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-16T19:37:11.946 INFO:teuthology.orchestra.run.smithi109.stdout:Discarding blocks...Done. 2021-09-16T19:37:11.947 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi109.front.sepia.ceph.com -o noatime 2021-09-16T19:37:11.948 DEBUG:teuthology.orchestra.run.smithi109:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2021-09-16T19:37:12.055 DEBUG:teuthology.orchestra.run.smithi109:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2021-09-16T19:37:12.082 DEBUG:teuthology.orchestra.run.smithi109:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-16T19:37:12.162 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:12.161+0000 7fefde7d50c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2021-09-16T19:37:12.164 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:12.162+0000 7fefde7d50c0 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2021-09-16T19:37:12.164 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:12.162+0000 7fefde7d50c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-16T19:37:12.165 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:12.162+0000 7fefde7d50c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-16T19:37:12.165 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:12.162+0000 7fefde7d50c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-16T19:37:12.165 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:12.163+0000 7fefde7d50c0 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2021-09-16T19:37:14.244 DEBUG:teuthology.orchestra.run.smithi109:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2021-09-16T19:37:14.313 DEBUG:teuthology.orchestra.run.smithi109:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-16T19:37:14.399 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:14.398+0000 7f887e7fc0c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2021-09-16T19:37:14.400 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:14.398+0000 7f887e7fc0c0 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2021-09-16T19:37:14.400 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:14.398+0000 7f887e7fc0c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-16T19:37:14.400 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:14.398+0000 7f887e7fc0c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-16T19:37:14.401 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:14.398+0000 7f887e7fc0c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-16T19:37:14.401 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:14.398+0000 7f887e7fc0c0 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2021-09-16T19:37:16.485 DEBUG:teuthology.orchestra.run.smithi109:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2021-09-16T19:37:16.513 DEBUG:teuthology.orchestra.run.smithi109:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-16T19:37:16.597 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:16.597+0000 7f2e6a8d30c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2021-09-16T19:37:16.598 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:16.597+0000 7f2e6a8d30c0 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2021-09-16T19:37:16.598 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:16.597+0000 7f2e6a8d30c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-16T19:37:16.598 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:16.597+0000 7f2e6a8d30c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-16T19:37:16.599 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:16.597+0000 7f2e6a8d30c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-16T19:37:16.599 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:37:16.597+0000 7f2e6a8d30c0 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2021-09-16T19:37:18.678 DEBUG:teuthology.orchestra.run.smithi109:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2021-09-16T19:37:18.706 INFO:tasks.ceph:Reading keys from all nodes... 2021-09-16T19:37:18.707 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:18.707 DEBUG:teuthology.orchestra.run.smithi109:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2021-09-16T19:37:18.773 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:18.773 DEBUG:teuthology.orchestra.run.smithi109:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2021-09-16T19:37:18.838 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:18.838 DEBUG:teuthology.orchestra.run.smithi109:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2021-09-16T19:37:18.903 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:18.903 DEBUG:teuthology.orchestra.run.smithi109:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2021-09-16T19:37:18.968 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:18.968 DEBUG:teuthology.orchestra.run.smithi109:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2021-09-16T19:37:19.023 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:19.024 DEBUG:teuthology.orchestra.run.smithi109:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2021-09-16T19:37:19.077 INFO:tasks.ceph:Adding keys to all mons... 2021-09-16T19:37:19.078 DEBUG:teuthology.orchestra.run.smithi109:> sudo tee -a /etc/ceph/c1.keyring 2021-09-16T19:37:19.141 INFO:teuthology.orchestra.run.smithi109.stdout:[mgr.x] 2021-09-16T19:37:19.141 INFO:teuthology.orchestra.run.smithi109.stdout: key = AQDlnENhV712OxAAw/krn3GHPZnrthvV4x0NUQ== 2021-09-16T19:37:19.141 INFO:teuthology.orchestra.run.smithi109.stdout:[osd.0] 2021-09-16T19:37:19.141 INFO:teuthology.orchestra.run.smithi109.stdout: key = AQDonENhFwirCRAAn2jz2zGP95jPZRkKRYBvvA== 2021-09-16T19:37:19.142 INFO:teuthology.orchestra.run.smithi109.stdout:[osd.1] 2021-09-16T19:37:19.142 INFO:teuthology.orchestra.run.smithi109.stdout: key = AQDqnENhp2bLFxAAUG/ncKw2mY8IidreQgBLKQ== 2021-09-16T19:37:19.142 INFO:teuthology.orchestra.run.smithi109.stdout:[osd.2] 2021-09-16T19:37:19.142 INFO:teuthology.orchestra.run.smithi109.stdout: key = AQDsnENhSJ2aIxAAFwrn11NpYLxOj18Qkp3aXQ== 2021-09-16T19:37:19.143 INFO:teuthology.orchestra.run.smithi109.stdout:[client.0] 2021-09-16T19:37:19.143 INFO:teuthology.orchestra.run.smithi109.stdout: key = AQDmnENhTytAAhAA6IHczkj0BZopiSrvKHw+wA== 2021-09-16T19:37:19.143 INFO:teuthology.orchestra.run.smithi109.stdout:[client.1] 2021-09-16T19:37:19.143 INFO:teuthology.orchestra.run.smithi109.stdout: key = AQDmnENhwEeQBxAAlryYGvDhJAnlqAbiSyBjkg== 2021-09-16T19:37:19.144 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-09-16T19:37:19.232 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-16T19:37:19.277 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-16T19:37:19.323 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-16T19:37:19.369 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-16T19:37:19.418 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-16T19:37:19.467 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-09-16T19:37:19.467 DEBUG:teuthology.orchestra.run.smithi109:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2021-09-16T19:37:19.491 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2021-09-16T19:37:19.584 DEBUG:teuthology.orchestra.run.smithi109:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2021-09-16T19:37:19.607 DEBUG:teuthology.orchestra.run.smithi109:> rm -- /home/ubuntu/cephtest/c1.monmap 2021-09-16T19:37:19.661 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2021-09-16T19:37:19.662 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2021-09-16T19:37:19.662 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2021-09-16T19:37:19.703 INFO:tasks.ceph.c1.mon.a:Started 2021-09-16T19:37:19.704 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2021-09-16T19:37:19.704 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2021-09-16T19:37:19.705 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2021-09-16T19:37:19.707 INFO:tasks.ceph.c1.mgr.x:Started 2021-09-16T19:37:19.708 INFO:tasks.ceph:Setting crush tunables to default 2021-09-16T19:37:19.708 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph --cluster c1 osd crush tunables default 2021-09-16T19:37:19.909 INFO:teuthology.orchestra.run.smithi109.stderr:adjusted tunables profile to default 2021-09-16T19:37:19.919 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2021-09-16T19:37:19.920 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:19.920 DEBUG:teuthology.orchestra.run.smithi109:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2021-09-16T19:37:19.944 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:19.945 DEBUG:teuthology.orchestra.run.smithi109:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2021-09-16T19:37:20.007 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:20.008 DEBUG:teuthology.orchestra.run.smithi109:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2021-09-16T19:37:20.072 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph --cluster c1 osd new edac18b9-28ea-4ebb-a46e-54b0fc0affbf 0 2021-09-16T19:37:20.331 INFO:teuthology.orchestra.run.smithi109.stdout:0 2021-09-16T19:37:20.343 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph --cluster c1 osd new 264854eb-ff3d-4c6a-a56e-0de6749f09f3 1 2021-09-16T19:37:20.552 INFO:teuthology.orchestra.run.smithi109.stdout:1 2021-09-16T19:37:20.566 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph --cluster c1 osd new 46e93252-d3d1-4ed8-ad88-8796328b9e04 2 2021-09-16T19:37:20.772 INFO:teuthology.orchestra.run.smithi109.stdout:2 2021-09-16T19:37:20.785 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2021-09-16T19:37:20.785 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2021-09-16T19:37:20.787 INFO:tasks.ceph.c1.osd.0:Started 2021-09-16T19:37:20.788 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2021-09-16T19:37:20.788 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2021-09-16T19:37:20.790 INFO:tasks.ceph.c1.osd.1:Started 2021-09-16T19:37:20.790 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2021-09-16T19:37:20.791 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2021-09-16T19:37:20.793 INFO:tasks.ceph.c1.osd.2:Started 2021-09-16T19:37:20.793 INFO:tasks.ceph:Waiting for OSDs to come up 2021-09-16T19:37:21.096 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-09-16T19:37:21.276 INFO:teuthology.misc.health.smithi109.stdout: 2021-09-16T19:37:21.276 INFO:teuthology.misc.health.smithi109.stdout:{"epoch":5,"fsid":"3c353c67-c60a-4c36-92c7-ea526a847dac","created":"2021-09-16T19:37:19.812711+0000","modified":"2021-09-16T19:37:20.765294+0000","last_up_change":"0.000000","last_in_change":"2021-09-16T19:37:20.765294+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"edac18b9-28ea-4ebb-a46e-54b0fc0affbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264854eb-ff3d-4c6a-a56e-0de6749f09f3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"46e93252-d3d1-4ed8-ad88-8796328b9e04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-16T19:37:21.287 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-09-16T19:37:21.385 INFO:tasks.ceph.c1.osd.0.smithi109.stderr:2021-09-16T19:37:21.383+0000 7f18315250c0 -1 Falling back to public interface 2021-09-16T19:37:21.385 INFO:tasks.ceph.c1.osd.1.smithi109.stderr:2021-09-16T19:37:21.383+0000 7f0e1e4690c0 -1 Falling back to public interface 2021-09-16T19:37:21.386 INFO:tasks.ceph.c1.osd.1.smithi109.stderr:2021-09-16T19:37:21.383+0000 7f0e1862e700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2021-09-16T19:37:21.390 INFO:tasks.ceph.c1.osd.2.smithi109.stderr:2021-09-16T19:37:21.389+0000 7fe62fcb70c0 -1 Falling back to public interface 2021-09-16T19:37:24.098 INFO:tasks.ceph.c1.osd.1.smithi109.stderr:2021-09-16T19:37:24.097+0000 7f0e1e4690c0 -1 osd.1 0 log_to_monitors {default=true} 2021-09-16T19:37:24.139 INFO:tasks.ceph.c1.osd.2.smithi109.stderr:2021-09-16T19:37:24.138+0000 7fe62fcb70c0 -1 osd.2 0 log_to_monitors {default=true} 2021-09-16T19:37:24.148 INFO:tasks.ceph.c1.osd.0.smithi109.stderr:2021-09-16T19:37:24.147+0000 7f18315250c0 -1 osd.0 0 log_to_monitors {default=true} 2021-09-16T19:37:26.330 INFO:tasks.ceph.c1.osd.0.smithi109.stderr:2021-09-16T19:37:26.329+0000 7f1827ee3700 -1 osd.0 0 waiting for initial osdmap 2021-09-16T19:37:26.332 INFO:tasks.ceph.c1.osd.0.smithi109.stderr:2021-09-16T19:37:26.331+0000 7f1820876700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-16T19:37:26.341 INFO:tasks.ceph.c1.osd.2.smithi109.stderr:2021-09-16T19:37:26.340+0000 7fe626675700 -1 osd.2 0 waiting for initial osdmap 2021-09-16T19:37:26.345 INFO:tasks.ceph.c1.osd.2.smithi109.stderr:2021-09-16T19:37:26.344+0000 7fe61f008700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-16T19:37:26.427 INFO:tasks.ceph.c1.osd.1.smithi109.stderr:2021-09-16T19:37:26.426+0000 7f0e14e27700 -1 osd.1 0 waiting for initial osdmap 2021-09-16T19:37:26.431 INFO:tasks.ceph.c1.osd.1.smithi109.stderr:2021-09-16T19:37:26.430+0000 7f0e0d7ba700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-16T19:37:27.591 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-09-16T19:37:27.882 INFO:teuthology.misc.health.smithi109.stdout: 2021-09-16T19:37:27.883 INFO:teuthology.misc.health.smithi109.stdout:{"epoch":8,"fsid":"3c353c67-c60a-4c36-92c7-ea526a847dac","created":"2021-09-16T19:37:19.812711+0000","modified":"2021-09-16T19:37:27.338138+0000","last_up_change":"2021-09-16T19:37:27.338138+0000","last_in_change":"2021-09-16T19:37:20.765294+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"edac18b9-28ea-4ebb-a46e-54b0fc0affbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6801","nonce":31407},{"type":"v1","addr":"172.21.15.109:6803","nonce":31407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6805","nonce":31407},{"type":"v1","addr":"172.21.15.109:6807","nonce":31407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6813","nonce":31407},{"type":"v1","addr":"172.21.15.109:6815","nonce":31407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6809","nonce":31407},{"type":"v1","addr":"172.21.15.109:6811","nonce":31407}]},"public_addr":"172.21.15.109:6803/31407","cluster_addr":"172.21.15.109:6807/31407","heartbeat_back_addr":"172.21.15.109:6815/31407","heartbeat_front_addr":"172.21.15.109:6811/31407","state":["exists","up"]},{"osd":1,"uuid":"264854eb-ff3d-4c6a-a56e-0de6749f09f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6800","nonce":31408},{"type":"v1","addr":"172.21.15.109:6802","nonce":31408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6804","nonce":31408},{"type":"v1","addr":"172.21.15.109:6806","nonce":31408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6812","nonce":31408},{"type":"v1","addr":"172.21.15.109:6814","nonce":31408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6808","nonce":31408},{"type":"v1","addr":"172.21.15.109:6810","nonce":31408}]},"public_addr":"172.21.15.109:6802/31408","cluster_addr":"172.21.15.109:6806/31408","heartbeat_back_addr":"172.21.15.109:6814/31408","heartbeat_front_addr":"172.21.15.109:6810/31408","state":["exists","up"]},{"osd":2,"uuid":"46e93252-d3d1-4ed8-ad88-8796328b9e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6816","nonce":31409},{"type":"v1","addr":"172.21.15.109:6817","nonce":31409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6818","nonce":31409},{"type":"v1","addr":"172.21.15.109:6819","nonce":31409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6822","nonce":31409},{"type":"v1","addr":"172.21.15.109:6823","nonce":31409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6820","nonce":31409},{"type":"v1","addr":"172.21.15.109:6821","nonce":31409}]},"public_addr":"172.21.15.109:6817/31409","cluster_addr":"172.21.15.109:6819/31409","heartbeat_back_addr":"172.21.15.109:6823/31409","heartbeat_front_addr":"172.21.15.109:6821/31409","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-16T19:37:27.893 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-09-16T19:37:27.894 INFO:tasks.ceph:Creating RBD pool 2021-09-16T19:37:27.894 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph --cluster c1 osd pool create rbd 8 2021-09-16T19:37:28.731 INFO:teuthology.orchestra.run.smithi109.stderr:pool 'rbd' created 2021-09-16T19:37:28.742 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-09-16T19:37:29.740 INFO:teuthology.orchestra.run.smithi109.stderr:enabled application 'rbd' on pool 'rbd' 2021-09-16T19:37:29.748 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2021-09-16T19:37:29.749 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-16T19:37:29.750 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-09-16T19:37:30.072 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:30.072 INFO:teuthology.orchestra.run.smithi109.stdout:{"epoch":10,"fsid":"3c353c67-c60a-4c36-92c7-ea526a847dac","created":"2021-09-16T19:37:19.812711+0000","modified":"2021-09-16T19:37:29.724551+0000","last_up_change":"2021-09-16T19:37:27.338138+0000","last_in_change":"2021-09-16T19:37:20.765294+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-16T19:37:28.193204+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-16T19:37:28.680642+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"edac18b9-28ea-4ebb-a46e-54b0fc0affbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6801","nonce":31407},{"type":"v1","addr":"172.21.15.109:6803","nonce":31407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6805","nonce":31407},{"type":"v1","addr":"172.21.15.109:6807","nonce":31407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6813","nonce":31407},{"type":"v1","addr":"172.21.15.109:6815","nonce":31407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6809","nonce":31407},{"type":"v1","addr":"172.21.15.109:6811","nonce":31407}]},"public_addr":"172.21.15.109:6803/31407","cluster_addr":"172.21.15.109:6807/31407","heartbeat_back_addr":"172.21.15.109:6815/31407","heartbeat_front_addr":"172.21.15.109:6811/31407","state":["exists","up"]},{"osd":1,"uuid":"264854eb-ff3d-4c6a-a56e-0de6749f09f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6800","nonce":31408},{"type":"v1","addr":"172.21.15.109:6802","nonce":31408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6804","nonce":31408},{"type":"v1","addr":"172.21.15.109:6806","nonce":31408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6812","nonce":31408},{"type":"v1","addr":"172.21.15.109:6814","nonce":31408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6808","nonce":31408},{"type":"v1","addr":"172.21.15.109:6810","nonce":31408}]},"public_addr":"172.21.15.109:6802/31408","cluster_addr":"172.21.15.109:6806/31408","heartbeat_back_addr":"172.21.15.109:6814/31408","heartbeat_front_addr":"172.21.15.109:6810/31408","state":["exists","up"]},{"osd":2,"uuid":"46e93252-d3d1-4ed8-ad88-8796328b9e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6816","nonce":31409},{"type":"v1","addr":"172.21.15.109:6817","nonce":31409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6818","nonce":31409},{"type":"v1","addr":"172.21.15.109:6819","nonce":31409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6822","nonce":31409},{"type":"v1","addr":"172.21.15.109:6823","nonce":31409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6820","nonce":31409},{"type":"v1","addr":"172.21.15.109:6821","nonce":31409}]},"public_addr":"172.21.15.109:6817/31409","cluster_addr":"172.21.15.109:6819/31409","heartbeat_back_addr":"172.21.15.109:6823/31409","heartbeat_front_addr":"172.21.15.109:6821/31409","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:26.330466+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:25.094609+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:25.148875+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-16T19:37:30.085 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-09-16T19:37:28.193204+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2021-09-16T19:37:28.680642+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2021-09-16T19:37:30.087 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get rbd pg_num 2021-09-16T19:37:30.382 INFO:teuthology.orchestra.run.smithi109.stdout:pg_num: 8 2021-09-16T19:37:30.397 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get .mgr pg_num 2021-09-16T19:37:30.684 INFO:teuthology.orchestra.run.smithi109.stdout:pg_num: 1 2021-09-16T19:37:30.697 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2021-09-16T19:37:30.698 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2021-09-16T19:37:30.698 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2021-09-16T19:37:31.026 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:31.038 INFO:teuthology.orchestra.run.smithi109.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6824","nonce":31087},{"type":"v1","addr":"172.21.15.109:6825","nonce":31087}]},"active_addr":"172.21.15.109:6825/31087","active_change":"2021-09-16T19:37:24.648929+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"Determines the behavior of the autoscaler algorithm `scale-up` means that it starts out with minmum pgs and scales up when there is pressure, `scale-down` means starts out with full pgs and scales down when there is pressure ","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share perf counter metrics summed across the whole cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.109:0","nonce":3876045118}]},{"addrvec":[{"type":"v2","addr":"172.21.15.109:0","nonce":4230255485}]},{"addrvec":[{"type":"v2","addr":"172.21.15.109:0","nonce":2373933028}]},{"addrvec":[{"type":"v2","addr":"172.21.15.109:0","nonce":2454211134}]}]}} 2021-09-16T19:37:31.040 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2021-09-16T19:37:31.041 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2021-09-16T19:37:31.041 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-16T19:37:31.326 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:31.327 INFO:teuthology.orchestra.run.smithi109.stdout:{"epoch":11,"fsid":"3c353c67-c60a-4c36-92c7-ea526a847dac","created":"2021-09-16T19:37:19.812711+0000","modified":"2021-09-16T19:37:30.731468+0000","last_up_change":"2021-09-16T19:37:27.338138+0000","last_in_change":"2021-09-16T19:37:20.765294+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-16T19:37:28.193204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-16T19:37:28.680642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"edac18b9-28ea-4ebb-a46e-54b0fc0affbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6801","nonce":31407},{"type":"v1","addr":"172.21.15.109:6803","nonce":31407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6805","nonce":31407},{"type":"v1","addr":"172.21.15.109:6807","nonce":31407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6813","nonce":31407},{"type":"v1","addr":"172.21.15.109:6815","nonce":31407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6809","nonce":31407},{"type":"v1","addr":"172.21.15.109:6811","nonce":31407}]},"public_addr":"172.21.15.109:6803/31407","cluster_addr":"172.21.15.109:6807/31407","heartbeat_back_addr":"172.21.15.109:6815/31407","heartbeat_front_addr":"172.21.15.109:6811/31407","state":["exists","up"]},{"osd":1,"uuid":"264854eb-ff3d-4c6a-a56e-0de6749f09f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6800","nonce":31408},{"type":"v1","addr":"172.21.15.109:6802","nonce":31408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6804","nonce":31408},{"type":"v1","addr":"172.21.15.109:6806","nonce":31408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6812","nonce":31408},{"type":"v1","addr":"172.21.15.109:6814","nonce":31408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6808","nonce":31408},{"type":"v1","addr":"172.21.15.109:6810","nonce":31408}]},"public_addr":"172.21.15.109:6802/31408","cluster_addr":"172.21.15.109:6806/31408","heartbeat_back_addr":"172.21.15.109:6814/31408","heartbeat_front_addr":"172.21.15.109:6810/31408","state":["exists","up"]},{"osd":2,"uuid":"46e93252-d3d1-4ed8-ad88-8796328b9e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6816","nonce":31409},{"type":"v1","addr":"172.21.15.109:6817","nonce":31409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6818","nonce":31409},{"type":"v1","addr":"172.21.15.109:6819","nonce":31409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6822","nonce":31409},{"type":"v1","addr":"172.21.15.109:6823","nonce":31409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6820","nonce":31409},{"type":"v1","addr":"172.21.15.109:6821","nonce":31409}]},"public_addr":"172.21.15.109:6817/31409","cluster_addr":"172.21.15.109:6819/31409","heartbeat_back_addr":"172.21.15.109:6823/31409","heartbeat_front_addr":"172.21.15.109:6821/31409","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:26.330466+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:25.094609+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:25.148875+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-16T19:37:31.339 INFO:tasks.ceph.ceph_manager.c1:all up! 2021-09-16T19:37:31.339 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-16T19:37:31.623 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:31.624 INFO:teuthology.orchestra.run.smithi109.stdout:{"epoch":11,"fsid":"3c353c67-c60a-4c36-92c7-ea526a847dac","created":"2021-09-16T19:37:19.812711+0000","modified":"2021-09-16T19:37:30.731468+0000","last_up_change":"2021-09-16T19:37:27.338138+0000","last_in_change":"2021-09-16T19:37:20.765294+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-16T19:37:28.193204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-16T19:37:28.680642+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"edac18b9-28ea-4ebb-a46e-54b0fc0affbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6801","nonce":31407},{"type":"v1","addr":"172.21.15.109:6803","nonce":31407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6805","nonce":31407},{"type":"v1","addr":"172.21.15.109:6807","nonce":31407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6813","nonce":31407},{"type":"v1","addr":"172.21.15.109:6815","nonce":31407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6809","nonce":31407},{"type":"v1","addr":"172.21.15.109:6811","nonce":31407}]},"public_addr":"172.21.15.109:6803/31407","cluster_addr":"172.21.15.109:6807/31407","heartbeat_back_addr":"172.21.15.109:6815/31407","heartbeat_front_addr":"172.21.15.109:6811/31407","state":["exists","up"]},{"osd":1,"uuid":"264854eb-ff3d-4c6a-a56e-0de6749f09f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6800","nonce":31408},{"type":"v1","addr":"172.21.15.109:6802","nonce":31408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6804","nonce":31408},{"type":"v1","addr":"172.21.15.109:6806","nonce":31408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6812","nonce":31408},{"type":"v1","addr":"172.21.15.109:6814","nonce":31408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6808","nonce":31408},{"type":"v1","addr":"172.21.15.109:6810","nonce":31408}]},"public_addr":"172.21.15.109:6802/31408","cluster_addr":"172.21.15.109:6806/31408","heartbeat_back_addr":"172.21.15.109:6814/31408","heartbeat_front_addr":"172.21.15.109:6810/31408","state":["exists","up"]},{"osd":2,"uuid":"46e93252-d3d1-4ed8-ad88-8796328b9e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6816","nonce":31409},{"type":"v1","addr":"172.21.15.109:6817","nonce":31409}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6818","nonce":31409},{"type":"v1","addr":"172.21.15.109:6819","nonce":31409}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6822","nonce":31409},{"type":"v1","addr":"172.21.15.109:6823","nonce":31409}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.109:6820","nonce":31409},{"type":"v1","addr":"172.21.15.109:6821","nonce":31409}]},"public_addr":"172.21.15.109:6817/31409","cluster_addr":"172.21.15.109:6819/31409","heartbeat_back_addr":"172.21.15.109:6823/31409","heartbeat_front_addr":"172.21.15.109:6821/31409","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:26.330466+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:25.094609+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:25.148875+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-16T19:37:31.637 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2021-09-16T19:37:31.781 INFO:teuthology.orchestra.run.smithi109.stdout:34359738372 2021-09-16T19:37:31.782 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2021-09-16T19:37:31.926 INFO:teuthology.orchestra.run.smithi109.stdout:34359738371 2021-09-16T19:37:31.926 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2021-09-16T19:37:32.069 INFO:teuthology.orchestra.run.smithi109.stdout:34359738372 2021-09-16T19:37:32.070 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-09-16T19:37:32.353 INFO:teuthology.orchestra.run.smithi109.stdout:34359738371 2021-09-16T19:37:32.366 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.0 2021-09-16T19:37:33.367 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-09-16T19:37:33.665 INFO:teuthology.orchestra.run.smithi109.stdout:34359738372 2021-09-16T19:37:33.678 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.0 2021-09-16T19:37:33.679 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2021-09-16T19:37:33.965 INFO:teuthology.orchestra.run.smithi109.stdout:34359738371 2021-09-16T19:37:33.978 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.1 2021-09-16T19:37:33.979 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2021-09-16T19:37:34.263 INFO:teuthology.orchestra.run.smithi109.stdout:34359738372 2021-09-16T19:37:34.276 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.2 2021-09-16T19:37:34.277 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2021-09-16T19:37:34.277 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-09-16T19:37:34.555 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:34.556 INFO:teuthology.orchestra.run.smithi109.stderr:dumped all 2021-09-16T19:37:34.567 INFO:teuthology.orchestra.run.smithi109.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2021-09-16T19:37:32.654003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24568,"kb_used_data":1440,"kb_used_omap":0,"kb_used_meta":23104,"kb_avail":314548232,"statfs":{"total":322122547200,"available":322097389568,"internally_reserved":0,"allocated":1474560,"data_stored":1255508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23658496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:29.739557+0000","last_change":"2021-09-16T19:37:29.735714+0000","last_active":"2021-09-16T19:37:29.739557+0000","last_peered":"2021-09-16T19:37:29.739557+0000","last_clean":"2021-09-16T19:37:29.739557+0000","last_became_active":"2021-09-16T19:37:29.735383+0000","last_became_peered":"2021-09-16T19:37:29.735383+0000","last_unstale":"2021-09-16T19:37:29.739557+0000","last_undegraded":"2021-09-16T19:37:29.739557+0000","last_fullsized":"2021-09-16T19:37:29.739557+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:29.742294+0000","last_change":"2021-09-16T19:37:29.736982+0000","last_active":"2021-09-16T19:37:29.742294+0000","last_peered":"2021-09-16T19:37:29.742294+0000","last_clean":"2021-09-16T19:37:29.742294+0000","last_became_active":"2021-09-16T19:37:29.736769+0000","last_became_peered":"2021-09-16T19:37:29.736769+0000","last_unstale":"2021-09-16T19:37:29.742294+0000","last_undegraded":"2021-09-16T19:37:29.742294+0000","last_fullsized":"2021-09-16T19:37:29.742294+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:29.738636+0000","last_change":"2021-09-16T19:37:29.737365+0000","last_active":"2021-09-16T19:37:29.738636+0000","last_peered":"2021-09-16T19:37:29.738636+0000","last_clean":"2021-09-16T19:37:29.738636+0000","last_became_active":"2021-09-16T19:37:29.737068+0000","last_became_peered":"2021-09-16T19:37:29.737068+0000","last_unstale":"2021-09-16T19:37:29.738636+0000","last_undegraded":"2021-09-16T19:37:29.738636+0000","last_fullsized":"2021-09-16T19:37:29.738636+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:29.742303+0000","last_change":"2021-09-16T19:37:29.737333+0000","last_active":"2021-09-16T19:37:29.742303+0000","last_peered":"2021-09-16T19:37:29.742303+0000","last_clean":"2021-09-16T19:37:29.742303+0000","last_became_active":"2021-09-16T19:37:29.737170+0000","last_became_peered":"2021-09-16T19:37:29.737170+0000","last_unstale":"2021-09-16T19:37:29.742303+0000","last_undegraded":"2021-09-16T19:37:29.742303+0000","last_fullsized":"2021-09-16T19:37:29.742303+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:29.736838+0000","last_change":"2021-09-16T19:37:29.735230+0000","last_active":"2021-09-16T19:37:29.736838+0000","last_peered":"2021-09-16T19:37:29.736838+0000","last_clean":"2021-09-16T19:37:29.736838+0000","last_became_active":"2021-09-16T19:37:29.734953+0000","last_became_peered":"2021-09-16T19:37:29.734953+0000","last_unstale":"2021-09-16T19:37:29.736838+0000","last_undegraded":"2021-09-16T19:37:29.736838+0000","last_fullsized":"2021-09-16T19:37:29.736838+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:30.209594+0000","last_change":"2021-09-16T19:37:29.738406+0000","last_active":"2021-09-16T19:37:30.209594+0000","last_peered":"2021-09-16T19:37:30.209594+0000","last_clean":"2021-09-16T19:37:30.209594+0000","last_became_active":"2021-09-16T19:37:29.737471+0000","last_became_peered":"2021-09-16T19:37:29.737471+0000","last_unstale":"2021-09-16T19:37:30.209594+0000","last_undegraded":"2021-09-16T19:37:30.209594+0000","last_fullsized":"2021-09-16T19:37:30.209594+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:29.739518+0000","last_change":"2021-09-16T19:37:29.735345+0000","last_active":"2021-09-16T19:37:29.739518+0000","last_peered":"2021-09-16T19:37:29.739518+0000","last_clean":"2021-09-16T19:37:29.739518+0000","last_became_active":"2021-09-16T19:37:29.735031+0000","last_became_peered":"2021-09-16T19:37:29.735031+0000","last_unstale":"2021-09-16T19:37:29.739518+0000","last_undegraded":"2021-09-16T19:37:29.739518+0000","last_fullsized":"2021-09-16T19:37:29.739518+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:29.742312+0000","last_change":"2021-09-16T19:37:29.737350+0000","last_active":"2021-09-16T19:37:29.742312+0000","last_peered":"2021-09-16T19:37:29.742312+0000","last_clean":"2021-09-16T19:37:29.742312+0000","last_became_active":"2021-09-16T19:37:29.737065+0000","last_became_peered":"2021-09-16T19:37:29.737065+0000","last_unstale":"2021-09-16T19:37:29.742312+0000","last_undegraded":"2021-09-16T19:37:29.742312+0000","last_fullsized":"2021-09-16T19:37:29.742312+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:29.738649+0000","last_change":"2021-09-16T19:37:29.737189+0000","last_active":"2021-09-16T19:37:29.738649+0000","last_peered":"2021-09-16T19:37:29.738649+0000","last_clean":"2021-09-16T19:37:29.738649+0000","last_became_active":"2021-09-16T19:37:29.736890+0000","last_became_peered":"2021-09-16T19:37:29.736890+0000","last_unstale":"2021-09-16T19:37:29.738649+0000","last_undegraded":"2021-09-16T19:37:29.738649+0000","last_fullsized":"2021-09-16T19:37:29.738649+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8492,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":104849108,"statfs":{"total":107374182400,"available":107365486592,"internally_reserved":0,"allocated":692224,"data_stored":616560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7520,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":7424,"kb_avail":104850080,"statfs":{"total":107374182400,"available":107366481920,"internally_reserved":0,"allocated":90112,"data_stored":22388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7602176},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8556,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104849044,"statfs":{"total":107374182400,"available":107365421056,"internally_reserved":0,"allocated":692224,"data_stored":616560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-16T19:37:34.568 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-09-16T19:37:34.848 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:34.850 INFO:teuthology.orchestra.run.smithi109.stderr:dumped all 2021-09-16T19:37:34.861 INFO:teuthology.orchestra.run.smithi109.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2021-09-16T19:37:34.654450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24640,"kb_used_data":1448,"kb_used_omap":0,"kb_used_meta":23168,"kb_avail":314548160,"statfs":{"total":322122547200,"available":322097315840,"internally_reserved":0,"allocated":1482752,"data_stored":1259312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23724032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000447"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:29.739557+0000","last_change":"2021-09-16T19:37:29.735714+0000","last_active":"2021-09-16T19:37:29.739557+0000","last_peered":"2021-09-16T19:37:29.739557+0000","last_clean":"2021-09-16T19:37:29.739557+0000","last_became_active":"2021-09-16T19:37:29.735383+0000","last_became_peered":"2021-09-16T19:37:29.735383+0000","last_unstale":"2021-09-16T19:37:29.739557+0000","last_undegraded":"2021-09-16T19:37:29.739557+0000","last_fullsized":"2021-09-16T19:37:29.739557+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:29.742294+0000","last_change":"2021-09-16T19:37:29.736982+0000","last_active":"2021-09-16T19:37:29.742294+0000","last_peered":"2021-09-16T19:37:29.742294+0000","last_clean":"2021-09-16T19:37:29.742294+0000","last_became_active":"2021-09-16T19:37:29.736769+0000","last_became_peered":"2021-09-16T19:37:29.736769+0000","last_unstale":"2021-09-16T19:37:29.742294+0000","last_undegraded":"2021-09-16T19:37:29.742294+0000","last_fullsized":"2021-09-16T19:37:29.742294+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:29.738636+0000","last_change":"2021-09-16T19:37:29.737365+0000","last_active":"2021-09-16T19:37:29.738636+0000","last_peered":"2021-09-16T19:37:29.738636+0000","last_clean":"2021-09-16T19:37:29.738636+0000","last_became_active":"2021-09-16T19:37:29.737068+0000","last_became_peered":"2021-09-16T19:37:29.737068+0000","last_unstale":"2021-09-16T19:37:29.738636+0000","last_undegraded":"2021-09-16T19:37:29.738636+0000","last_fullsized":"2021-09-16T19:37:29.738636+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:29.742303+0000","last_change":"2021-09-16T19:37:29.737333+0000","last_active":"2021-09-16T19:37:29.742303+0000","last_peered":"2021-09-16T19:37:29.742303+0000","last_clean":"2021-09-16T19:37:29.742303+0000","last_became_active":"2021-09-16T19:37:29.737170+0000","last_became_peered":"2021-09-16T19:37:29.737170+0000","last_unstale":"2021-09-16T19:37:29.742303+0000","last_undegraded":"2021-09-16T19:37:29.742303+0000","last_fullsized":"2021-09-16T19:37:29.742303+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:29.736838+0000","last_change":"2021-09-16T19:37:29.735230+0000","last_active":"2021-09-16T19:37:29.736838+0000","last_peered":"2021-09-16T19:37:29.736838+0000","last_clean":"2021-09-16T19:37:29.736838+0000","last_became_active":"2021-09-16T19:37:29.734953+0000","last_became_peered":"2021-09-16T19:37:29.734953+0000","last_unstale":"2021-09-16T19:37:29.736838+0000","last_undegraded":"2021-09-16T19:37:29.736838+0000","last_fullsized":"2021-09-16T19:37:29.736838+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"10'97","reported_seq":136,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:30.209594+0000","last_change":"2021-09-16T19:37:29.738406+0000","last_active":"2021-09-16T19:37:30.209594+0000","last_peered":"2021-09-16T19:37:30.209594+0000","last_clean":"2021-09-16T19:37:30.209594+0000","last_became_active":"2021-09-16T19:37:29.737471+0000","last_became_peered":"2021-09-16T19:37:29.737471+0000","last_unstale":"2021-09-16T19:37:30.209594+0000","last_undegraded":"2021-09-16T19:37:30.209594+0000","last_fullsized":"2021-09-16T19:37:30.209594+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:29.739518+0000","last_change":"2021-09-16T19:37:29.735345+0000","last_active":"2021-09-16T19:37:29.739518+0000","last_peered":"2021-09-16T19:37:29.739518+0000","last_clean":"2021-09-16T19:37:29.739518+0000","last_became_active":"2021-09-16T19:37:29.735031+0000","last_became_peered":"2021-09-16T19:37:29.735031+0000","last_unstale":"2021-09-16T19:37:29.739518+0000","last_undegraded":"2021-09-16T19:37:29.739518+0000","last_fullsized":"2021-09-16T19:37:29.739518+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:29.742312+0000","last_change":"2021-09-16T19:37:29.737350+0000","last_active":"2021-09-16T19:37:29.742312+0000","last_peered":"2021-09-16T19:37:29.742312+0000","last_clean":"2021-09-16T19:37:29.742312+0000","last_became_active":"2021-09-16T19:37:29.737065+0000","last_became_peered":"2021-09-16T19:37:29.737065+0000","last_unstale":"2021-09-16T19:37:29.742312+0000","last_undegraded":"2021-09-16T19:37:29.742312+0000","last_fullsized":"2021-09-16T19:37:29.742312+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2021-09-16T19:37:29.738649+0000","last_change":"2021-09-16T19:37:29.737189+0000","last_active":"2021-09-16T19:37:29.738649+0000","last_peered":"2021-09-16T19:37:29.738649+0000","last_clean":"2021-09-16T19:37:29.738649+0000","last_became_active":"2021-09-16T19:37:29.736890+0000","last_became_peered":"2021-09-16T19:37:29.736890+0000","last_unstale":"2021-09-16T19:37:29.738649+0000","last_undegraded":"2021-09-16T19:37:29.738649+0000","last_fullsized":"2021-09-16T19:37:29.738649+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:28.721776+0000","last_clean_scrub_stamp":"2021-09-16T19:37:28.721776+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8492,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":104849108,"statfs":{"total":107374182400,"available":107365486592,"internally_reserved":0,"allocated":692224,"data_stored":616560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7592,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850008,"statfs":{"total":107374182400,"available":107366408192,"internally_reserved":0,"allocated":98304,"data_stored":26192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8556,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104849044,"statfs":{"total":107374182400,"available":107365421056,"internally_reserved":0,"allocated":692224,"data_stored":616560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-16T19:37:34.863 INFO:tasks.ceph.ceph_manager.c1:clean! 2021-09-16T19:37:34.863 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2021-09-16T19:37:34.863 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2021-09-16T19:37:34.864 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2021-09-16T19:37:35.204 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:35.205 INFO:teuthology.orchestra.run.smithi109.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-09-16T19:37:35.218 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2021-09-16T19:37:35.218 INFO:teuthology.run_tasks:Running task ceph... 2021-09-16T19:37:35.230 INFO:tasks.ceph:Creating ceph cluster c2... 2021-09-16T19:37:35.230 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-09-16T19:37:35.231 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mbenjamin-2021-09-16_15:12:17-rgw-wip-rgwadminops-fsid-distro-basic-smithi/6393866', 'branch': 'wip-rgwadminops-fsid', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '6393866', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'mbenjamin-2021-09-16_15:12:17-rgw-wip-rgwadminops-fsid-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-rgwadminops-fsid'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'bc749ef43279601f3e9305d4ad8723d440abaa66', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': 'bc749ef43279601f3e9305d4ad8723d440abaa66'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-rgw-sts-op-logs-plus-perl', 'sha1': '4e2e64ee99b6cbb34351964aad0f9df4874dea6d'}}, 'owner': 'scheduled_mbenjamin@shatner', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': 'bc749ef43279601f3e9305d4ad8723d440abaa66', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-rgw-sts-op-logs-plus-perl', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_4e2e64ee99b6cbb34351964aad0f9df4874dea6d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '4e2e64ee99b6cbb34351964aad0f9df4874dea6d', 'targets': {'smithi109.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDit7nKIpkndJnsa95XTFOPPxPMn5UoSxviofbE8fdv6otd42BKcm+tP55ory6N3+MkrJLaUuqFz++RRoIZKIWDcrH84eNqA4+6QouaQAgu+pE/Up1idQaX1XOmi+sPHX4sZTmcUJD1EAgS5otgTbEcMvcjdsA1HCJTSXjjBYyOFbiiw0GTgCn3ur6vmRzf55VaJX4KQYJc5ncAxgHHtVZmtUmKNRAbNqRWU0dALAwPbZPMXDoiL06z2mMoOQiEOPB9HeNfTuiumQhor88FAuB4QwswCJIUn7kWcdd+Q50B8C2xcULqP19KDmOPzZ6VQ+k5D2oMzY3hXf5NCrXZBb4OyOm8kiG/L2T4dva3ZA0TY/Gd7+dTeiob9BbjfRCf/aSc5QiZSSp5hcf2iUdcP1EeaDW6bNvp7A1u2QrP7truKxgtXs1J6w62p4JPwPuDGBWj6iN6wgVSmh7Cl6wnLGkgXJS3BA6DvvIQVZ4Mm/q5sTNPXajC8qAjF7gOaQXhO4s=', 'smithi118.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/mv+u+EjO7vwGqKRrKc1/4hE/xmnvsVd0Dz7wFkd88PDWJbemZ0yiW25IgARYTm1yKTAsbsE6T9o8nL9St/CSjrFYHTLfgL4e1XJ7JeNSOFEGMLye3FRvMZ+fBC3rYHO9i6zAP9PGrBtAxXfnx6TxfrPHuYM/uvHskbIccY0RgCC8ZtBtH5Ank0H+/eRfabQU0KlqamP1bvV7gDoSkhaGM7WJg+q9THZC0j3zvaN+q7fOQnjjvUPyClYttfIx8bYhaG7yGfxxuLzX+8bnQWJE4x5Hz2/aUl+9wgwvuK68Bt8mZebdgTj6gzWDZ9FEn5bfSTwXfP5rQmyqQt95CcEZHdS7KKFTP9tN8dwaJVxZDtbqqh/PsfuOo7L1ZDZQUINeGyV0qozcyHbM2O+GKMN4dGPCj2ByfmhAwn6ktXmQ/HOeNbZ8jTVpodUurhwKCd2OU2+dR1Y/Zu5m94V1g6ona4db93N+Z1l3Ry43ou9WOE7bwdhTGYNzeGznkFdq8lM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'bc749ef43279601f3e9305d4ad8723d440abaa66', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'bc749ef43279601f3e9305d4ad8723d440abaa66', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '63133ca92c299bd23f40f94d370cf5381b0b1313', 'timestamp': '2021-09-16_15:12:17', 'tube': 'smithi', 'user': 'mbenjamin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.30581'} 2021-09-16T19:37:35.231 DEBUG:teuthology.orchestra.run.smithi109:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-09-16T19:37:35.234 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-09-16T19:37:35.252 DEBUG:teuthology.orchestra.run.smithi109:> sudo install -d -m0777 -- /var/run/ceph 2021-09-16T19:37:35.288 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/run/ceph 2021-09-16T19:37:35.321 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:35.322 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/scratch_devs of=/dev/stdout 2021-09-16T19:37:35.378 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-09-16T19:37:35.379 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-09-16T19:37:35.437 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-09-16T19:37:35.437 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-16T19:37:35.437 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 111400 Links: 1 2021-09-16T19:37:35.438 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-16T19:37:35.438 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2021-09-16T19:37:35.438 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2021-09-16 19:36:26.142047331 +0000 2021-09-16T19:37:35.438 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2021-09-16 19:27:25.824530860 +0000 2021-09-16T19:37:35.439 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2021-09-16 19:27:25.824530860 +0000 2021-09-16T19:37:35.439 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2021-09-16T19:37:35.445 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2021-09-16T19:37:35.445 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2021-09-16T19:37:35.446 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000125863 s, 4.1 MB/s 2021-09-16T19:37:35.448 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-09-16T19:37:35.504 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-09-16T19:37:35.504 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-16T19:37:35.505 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 116913 Links: 1 2021-09-16T19:37:35.505 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-16T19:37:35.505 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2021-09-16T19:37:35.505 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2021-09-16 19:36:26.142047331 +0000 2021-09-16T19:37:35.506 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2021-09-16 19:27:26.175523013 +0000 2021-09-16T19:37:35.506 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2021-09-16 19:27:26.175523013 +0000 2021-09-16T19:37:35.506 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2021-09-16T19:37:35.514 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2021-09-16T19:37:35.515 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2021-09-16T19:37:35.515 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 8.9715e-05 s, 5.7 MB/s 2021-09-16T19:37:35.517 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-09-16T19:37:35.573 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-09-16T19:37:35.573 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-16T19:37:35.574 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 112373 Links: 1 2021-09-16T19:37:35.574 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-16T19:37:35.574 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2021-09-16T19:37:35.574 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2021-09-16 19:36:26.142047331 +0000 2021-09-16T19:37:35.575 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2021-09-16 19:27:26.514515438 +0000 2021-09-16T19:37:35.575 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2021-09-16 19:27:26.514515438 +0000 2021-09-16T19:37:35.575 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2021-09-16T19:37:35.583 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2021-09-16T19:37:35.583 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2021-09-16T19:37:35.583 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000111918 s, 4.6 MB/s 2021-09-16T19:37:35.585 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-09-16T19:37:35.640 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-09-16T19:37:35.641 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-16T19:37:35.641 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 6h/6d Inode: 115174 Links: 1 2021-09-16T19:37:35.641 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-16T19:37:35.641 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2021-09-16T19:37:35.641 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2021-09-16 19:36:26.142047331 +0000 2021-09-16T19:37:35.642 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2021-09-16 19:27:26.855507817 +0000 2021-09-16T19:37:35.642 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2021-09-16 19:27:26.855507817 +0000 2021-09-16T19:37:35.642 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2021-09-16T19:37:35.650 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2021-09-16T19:37:35.650 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2021-09-16T19:37:35.650 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000110916 s, 4.6 MB/s 2021-09-16T19:37:35.653 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-16T19:37:35.653 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-16T19:37:35.653 INFO:tasks.ceph:Generating config... 2021-09-16T19:37:35.655 INFO:tasks.ceph:[client] debug rgw = 20 2021-09-16T19:37:35.655 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-09-16T19:37:35.655 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-09-16T19:37:35.656 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-16T19:37:35.656 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-09-16T19:37:35.656 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-09-16T19:37:35.656 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-09-16T19:37:35.657 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-09-16T19:37:35.657 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2021-09-16T19:37:35.657 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-09-16T19:37:35.658 INFO:tasks.ceph:[client] setgroup = ceph 2021-09-16T19:37:35.658 INFO:tasks.ceph:[client] setuser = ceph 2021-09-16T19:37:35.658 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-09-16T19:37:35.658 INFO:tasks.ceph:[mgr] debug ms = 1 2021-09-16T19:37:35.659 INFO:tasks.ceph:[mon] debug mon = 20 2021-09-16T19:37:35.659 INFO:tasks.ceph:[mon] debug ms = 1 2021-09-16T19:37:35.659 INFO:tasks.ceph:[mon] debug paxos = 20 2021-09-16T19:37:35.660 INFO:tasks.ceph:[osd] debug ms = 1 2021-09-16T19:37:35.660 INFO:tasks.ceph:[osd] debug osd = 20 2021-09-16T19:37:35.660 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-09-16T19:37:35.662 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-09-16T19:37:35.662 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-09-16T19:37:35.662 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-09-16T19:37:35.663 INFO:tasks.ceph:Setting up c2.mon.a... 2021-09-16T19:37:35.663 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2021-09-16T19:37:35.761 INFO:teuthology.orchestra.run.smithi118.stdout:creating /etc/ceph/c2.keyring 2021-09-16T19:37:35.763 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2021-09-16T19:37:35.802 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-16T19:37:35.866 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.118')] 2021-09-16T19:37:35.867 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.118'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-09-16T19:37:35.868 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:35.869 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-09-16T19:37:35.921 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.118 --print /home/ubuntu/cephtest/c2.monmap 2021-09-16T19:37:35.995 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:37:35.995 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:37:35.997 INFO:teuthology.orchestra.run.smithi118.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2021-09-16T19:37:35.997 INFO:teuthology.orchestra.run.smithi118.stdout:monmaptool: generated fsid 64e7c9b4-a739-4726-94e9-afe7eeb05b8f 2021-09-16T19:37:35.997 INFO:teuthology.orchestra.run.smithi118.stdout:setting min_mon_release = octopus 2021-09-16T19:37:35.997 INFO:teuthology.orchestra.run.smithi118.stdout:epoch 0 2021-09-16T19:37:35.998 INFO:teuthology.orchestra.run.smithi118.stdout:fsid 64e7c9b4-a739-4726-94e9-afe7eeb05b8f 2021-09-16T19:37:35.998 INFO:teuthology.orchestra.run.smithi118.stdout:last_changed 2021-09-16T19:37:35.995696+0000 2021-09-16T19:37:35.998 INFO:teuthology.orchestra.run.smithi118.stdout:created 2021-09-16T19:37:35.995696+0000 2021-09-16T19:37:35.998 INFO:teuthology.orchestra.run.smithi118.stdout:min_mon_release 15 (octopus) 2021-09-16T19:37:35.999 INFO:teuthology.orchestra.run.smithi118.stdout:election_strategy: 1 2021-09-16T19:37:35.999 INFO:teuthology.orchestra.run.smithi118.stdout:0: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.a 2021-09-16T19:37:35.999 INFO:teuthology.orchestra.run.smithi118.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2021-09-16T19:37:36.000 DEBUG:teuthology.orchestra.run.smithi118:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-09-16T19:37:36.052 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 64e7c9b4-a739-4726-94e9-afe7eeb05b8f... 2021-09-16T19:37:36.053 DEBUG:teuthology.orchestra.run.smithi109:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-09-16T19:37:36.055 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-09-16T19:37:36.100 INFO:teuthology.orchestra.run.smithi109.stdout:[global] 2021-09-16T19:37:36.101 INFO:teuthology.orchestra.run.smithi109.stdout: chdir = "" 2021-09-16T19:37:36.101 INFO:teuthology.orchestra.run.smithi109.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-16T19:37:36.102 INFO:teuthology.orchestra.run.smithi109.stdout: auth supported = cephx 2021-09-16T19:37:36.102 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.103 INFO:teuthology.orchestra.run.smithi109.stdout: filestore xattr use omap = true 2021-09-16T19:37:36.103 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.104 INFO:teuthology.orchestra.run.smithi109.stdout: mon clock drift allowed = 1.000 2021-09-16T19:37:36.104 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.104 INFO:teuthology.orchestra.run.smithi109.stdout: osd crush chooseleaf type = 0 2021-09-16T19:37:36.104 INFO:teuthology.orchestra.run.smithi109.stdout: auth debug = true 2021-09-16T19:37:36.105 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.105 INFO:teuthology.orchestra.run.smithi109.stdout: ms die on old message = true 2021-09-16T19:37:36.105 INFO:teuthology.orchestra.run.smithi109.stdout: ms die on bug = true 2021-09-16T19:37:36.105 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.106 INFO:teuthology.orchestra.run.smithi109.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-16T19:37:36.106 INFO:teuthology.orchestra.run.smithi109.stdout: mon pg warn max object skew = 0 2021-09-16T19:37:36.106 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.106 INFO:teuthology.orchestra.run.smithi109.stdout: # disable pg_autoscaler by default for new pools 2021-09-16T19:37:36.107 INFO:teuthology.orchestra.run.smithi109.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-16T19:37:36.107 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.107 INFO:teuthology.orchestra.run.smithi109.stdout: osd pool default size = 2 2021-09-16T19:37:36.107 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.107 INFO:teuthology.orchestra.run.smithi109.stdout: mon osd allow primary affinity = true 2021-09-16T19:37:36.108 INFO:teuthology.orchestra.run.smithi109.stdout: mon osd allow pg remap = true 2021-09-16T19:37:36.108 INFO:teuthology.orchestra.run.smithi109.stdout: mon warn on legacy crush tunables = false 2021-09-16T19:37:36.108 INFO:teuthology.orchestra.run.smithi109.stdout: mon warn on crush straw calc version zero = false 2021-09-16T19:37:36.108 INFO:teuthology.orchestra.run.smithi109.stdout: mon warn on no sortbitwise = false 2021-09-16T19:37:36.108 INFO:teuthology.orchestra.run.smithi109.stdout: mon warn on osd down out interval zero = false 2021-09-16T19:37:36.109 INFO:teuthology.orchestra.run.smithi109.stdout: mon warn on too few osds = false 2021-09-16T19:37:36.109 INFO:teuthology.orchestra.run.smithi109.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-16T19:37:36.109 INFO:teuthology.orchestra.run.smithi109.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-16T19:37:36.109 INFO:teuthology.orchestra.run.smithi109.stdout: mon_allow_pool_size_one = true 2021-09-16T19:37:36.110 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.110 INFO:teuthology.orchestra.run.smithi109.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-16T19:37:36.110 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.110 INFO:teuthology.orchestra.run.smithi109.stdout: osd default data pool replay window = 5 2021-09-16T19:37:36.110 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.111 INFO:teuthology.orchestra.run.smithi109.stdout: mon allow pool delete = true 2021-09-16T19:37:36.111 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.111 INFO:teuthology.orchestra.run.smithi109.stdout: mon cluster log file level = debug 2021-09-16T19:37:36.111 INFO:teuthology.orchestra.run.smithi109.stdout: debug asserts on shutdown = true 2021-09-16T19:37:36.112 INFO:teuthology.orchestra.run.smithi109.stdout: mon health detail to clog = false 2021-09-16T19:37:36.112 INFO:teuthology.orchestra.run.smithi109.stdout: mon host = 172.21.15.118 2021-09-16T19:37:36.112 INFO:teuthology.orchestra.run.smithi109.stdout: fsid = 64e7c9b4-a739-4726-94e9-afe7eeb05b8f 2021-09-16T19:37:36.112 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.112 INFO:teuthology.orchestra.run.smithi109.stdout:[osd] 2021-09-16T19:37:36.113 INFO:teuthology.orchestra.run.smithi109.stdout: osd journal size = 100 2021-09-16T19:37:36.113 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.113 INFO:teuthology.orchestra.run.smithi109.stdout: osd scrub load threshold = 5.0 2021-09-16T19:37:36.113 INFO:teuthology.orchestra.run.smithi109.stdout: osd scrub max interval = 600 2021-09-16T19:37:36.114 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.114 INFO:teuthology.orchestra.run.smithi109.stdout: osd recover clone overlap = true 2021-09-16T19:37:36.114 INFO:teuthology.orchestra.run.smithi109.stdout: osd recovery max chunk = 1048576 2021-09-16T19:37:36.114 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.115 INFO:teuthology.orchestra.run.smithi109.stdout: osd debug shutdown = true 2021-09-16T19:37:36.115 INFO:teuthology.orchestra.run.smithi109.stdout: osd debug op order = true 2021-09-16T19:37:36.115 INFO:teuthology.orchestra.run.smithi109.stdout: osd debug verify stray on activate = true 2021-09-16T19:37:36.115 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.116 INFO:teuthology.orchestra.run.smithi109.stdout: osd open classes on start = true 2021-09-16T19:37:36.116 INFO:teuthology.orchestra.run.smithi109.stdout: osd debug pg log writeout = true 2021-09-16T19:37:36.116 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.116 INFO:teuthology.orchestra.run.smithi109.stdout: osd deep scrub update digest min age = 30 2021-09-16T19:37:36.116 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.117 INFO:teuthology.orchestra.run.smithi109.stdout: osd map max advance = 10 2021-09-16T19:37:36.117 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.117 INFO:teuthology.orchestra.run.smithi109.stdout: journal zero on create = true 2021-09-16T19:37:36.117 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.117 INFO:teuthology.orchestra.run.smithi109.stdout: filestore ondisk finisher threads = 3 2021-09-16T19:37:36.118 INFO:teuthology.orchestra.run.smithi109.stdout: filestore apply finisher threads = 3 2021-09-16T19:37:36.118 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.118 INFO:teuthology.orchestra.run.smithi109.stdout: bdev debug aio = true 2021-09-16T19:37:36.118 INFO:teuthology.orchestra.run.smithi109.stdout: osd debug misdirected ops = true 2021-09-16T19:37:36.119 INFO:teuthology.orchestra.run.smithi109.stdout: debug ms = 1 2021-09-16T19:37:36.119 INFO:teuthology.orchestra.run.smithi109.stdout: debug osd = 20 2021-09-16T19:37:36.119 INFO:teuthology.orchestra.run.smithi109.stdout: rocksdb delete range threshold = 0 2021-09-16T19:37:36.119 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.119 INFO:teuthology.orchestra.run.smithi109.stdout:[mgr] 2021-09-16T19:37:36.120 INFO:teuthology.orchestra.run.smithi109.stdout: debug ms = 1 2021-09-16T19:37:36.120 INFO:teuthology.orchestra.run.smithi109.stdout: debug mgr = 20 2021-09-16T19:37:36.120 INFO:teuthology.orchestra.run.smithi109.stdout: debug mon = 20 2021-09-16T19:37:36.120 INFO:teuthology.orchestra.run.smithi109.stdout: debug auth = 20 2021-09-16T19:37:36.121 INFO:teuthology.orchestra.run.smithi109.stdout: mon reweight min pgs per osd = 4 2021-09-16T19:37:36.121 INFO:teuthology.orchestra.run.smithi109.stdout: mon reweight min bytes per osd = 10 2021-09-16T19:37:36.121 INFO:teuthology.orchestra.run.smithi109.stdout: mgr/telemetry/nag = false 2021-09-16T19:37:36.121 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.121 INFO:teuthology.orchestra.run.smithi109.stdout:[mon] 2021-09-16T19:37:36.122 INFO:teuthology.orchestra.run.smithi109.stdout: debug ms = 1 2021-09-16T19:37:36.122 INFO:teuthology.orchestra.run.smithi109.stdout: debug mon = 20 2021-09-16T19:37:36.122 INFO:teuthology.orchestra.run.smithi109.stdout: debug paxos = 20 2021-09-16T19:37:36.122 INFO:teuthology.orchestra.run.smithi109.stdout: debug auth = 20 2021-09-16T19:37:36.123 INFO:teuthology.orchestra.run.smithi109.stdout: mon data avail warn = 5 2021-09-16T19:37:36.123 INFO:teuthology.orchestra.run.smithi109.stdout: mon mgr mkfs grace = 240 2021-09-16T19:37:36.123 INFO:teuthology.orchestra.run.smithi109.stdout: mon reweight min pgs per osd = 4 2021-09-16T19:37:36.123 INFO:teuthology.orchestra.run.smithi109.stdout: mon osd reporter subtree level = osd 2021-09-16T19:37:36.123 INFO:teuthology.orchestra.run.smithi109.stdout: mon osd prime pg temp = true 2021-09-16T19:37:36.124 INFO:teuthology.orchestra.run.smithi109.stdout: mon reweight min bytes per osd = 10 2021-09-16T19:37:36.124 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.124 INFO:teuthology.orchestra.run.smithi109.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-16T19:37:36.124 INFO:teuthology.orchestra.run.smithi109.stdout: auth mon ticket ttl = 660 # 11m 2021-09-16T19:37:36.124 INFO:teuthology.orchestra.run.smithi109.stdout: auth service ticket ttl = 240 # 4m 2021-09-16T19:37:36.125 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.125 INFO:teuthology.orchestra.run.smithi109.stdout: # don't complain about insecure global_id in the test suite 2021-09-16T19:37:36.125 INFO:teuthology.orchestra.run.smithi109.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-16T19:37:36.125 INFO:teuthology.orchestra.run.smithi109.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-16T19:37:36.126 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:37:36.126 INFO:teuthology.orchestra.run.smithi109.stdout:[client] 2021-09-16T19:37:36.126 INFO:teuthology.orchestra.run.smithi109.stdout: rgw cache enabled = true 2021-09-16T19:37:36.126 INFO:teuthology.orchestra.run.smithi109.stdout: rgw enable ops log = true 2021-09-16T19:37:36.126 INFO:teuthology.orchestra.run.smithi109.stdout: rgw enable usage log = true 2021-09-16T19:37:36.127 INFO:teuthology.orchestra.run.smithi109.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-16T19:37:36.127 INFO:teuthology.orchestra.run.smithi109.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-16T19:37:36.127 INFO:teuthology.orchestra.run.smithi109.stdout: debug rgw = 20 2021-09-16T19:37:36.127 INFO:teuthology.orchestra.run.smithi109.stdout: rgw crypt require ssl = False 2021-09-16T19:37:36.128 INFO:teuthology.orchestra.run.smithi109.stdout: rgw crypt s3 kms backend = testing 2021-09-16T19:37:36.128 INFO:teuthology.orchestra.run.smithi109.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-16T19:37:36.128 INFO:teuthology.orchestra.run.smithi109.stdout: rgw curl low speed time = 300 2021-09-16T19:37:36.128 INFO:teuthology.orchestra.run.smithi109.stdout: rgw data log num shards = 4 2021-09-16T19:37:36.128 INFO:teuthology.orchestra.run.smithi109.stdout: rgw md log max shards = 4 2021-09-16T19:37:36.129 INFO:teuthology.orchestra.run.smithi109.stdout: rgw sync log trim interval = 0 2021-09-16T19:37:36.129 INFO:teuthology.orchestra.run.smithi109.stdout: rgw sync meta inject err probability = 0.1 2021-09-16T19:37:36.129 INFO:teuthology.orchestra.run.smithi109.stdout: rgw sync obj etag verify = True 2021-09-16T19:37:36.129 INFO:teuthology.orchestra.run.smithi109.stdout: setgroup = ceph 2021-09-16T19:37:36.129 INFO:teuthology.orchestra.run.smithi109.stdout: setuser = ceph 2021-09-16T19:37:36.130 INFO:teuthology.orchestra.run.smithi109.stdout:[mon.a] 2021-09-16T19:37:36.130 INFO:teuthology.orchestra.run.smithi109.stdout:[osd.0] 2021-09-16T19:37:36.130 INFO:teuthology.orchestra.run.smithi109.stdout: osd_max_omap_entries_per_request = 10 2021-09-16T19:37:36.130 INFO:teuthology.orchestra.run.smithi109.stdout:[osd.1] 2021-09-16T19:37:36.131 INFO:teuthology.orchestra.run.smithi109.stdout: osd_max_omap_entries_per_request = 1000 2021-09-16T19:37:36.131 INFO:teuthology.orchestra.run.smithi109.stdout:[osd.2] 2021-09-16T19:37:36.131 INFO:teuthology.orchestra.run.smithi109.stdout: osd_max_omap_entries_per_request = 10000 2021-09-16T19:37:36.134 INFO:teuthology.orchestra.run.smithi118.stdout:[global] 2021-09-16T19:37:36.134 INFO:teuthology.orchestra.run.smithi118.stdout: chdir = "" 2021-09-16T19:37:36.134 INFO:teuthology.orchestra.run.smithi118.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-16T19:37:36.135 INFO:teuthology.orchestra.run.smithi118.stdout: auth supported = cephx 2021-09-16T19:37:36.135 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.135 INFO:teuthology.orchestra.run.smithi118.stdout: filestore xattr use omap = true 2021-09-16T19:37:36.135 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.135 INFO:teuthology.orchestra.run.smithi118.stdout: mon clock drift allowed = 1.000 2021-09-16T19:37:36.136 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.136 INFO:teuthology.orchestra.run.smithi118.stdout: osd crush chooseleaf type = 0 2021-09-16T19:37:36.136 INFO:teuthology.orchestra.run.smithi118.stdout: auth debug = true 2021-09-16T19:37:36.136 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.137 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on old message = true 2021-09-16T19:37:36.137 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on bug = true 2021-09-16T19:37:36.137 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.137 INFO:teuthology.orchestra.run.smithi118.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-16T19:37:36.137 INFO:teuthology.orchestra.run.smithi118.stdout: mon pg warn max object skew = 0 2021-09-16T19:37:36.138 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.138 INFO:teuthology.orchestra.run.smithi118.stdout: # disable pg_autoscaler by default for new pools 2021-09-16T19:37:36.138 INFO:teuthology.orchestra.run.smithi118.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-16T19:37:36.139 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.139 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default size = 2 2021-09-16T19:37:36.139 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.139 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow primary affinity = true 2021-09-16T19:37:36.140 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow pg remap = true 2021-09-16T19:37:36.140 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on legacy crush tunables = false 2021-09-16T19:37:36.140 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on crush straw calc version zero = false 2021-09-16T19:37:36.140 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on no sortbitwise = false 2021-09-16T19:37:36.140 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on osd down out interval zero = false 2021-09-16T19:37:36.141 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on too few osds = false 2021-09-16T19:37:36.141 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-16T19:37:36.141 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-16T19:37:36.141 INFO:teuthology.orchestra.run.smithi118.stdout: mon_allow_pool_size_one = true 2021-09-16T19:37:36.142 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.142 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-16T19:37:36.142 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.142 INFO:teuthology.orchestra.run.smithi118.stdout: osd default data pool replay window = 5 2021-09-16T19:37:36.143 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.143 INFO:teuthology.orchestra.run.smithi118.stdout: mon allow pool delete = true 2021-09-16T19:37:36.143 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.143 INFO:teuthology.orchestra.run.smithi118.stdout: mon cluster log file level = debug 2021-09-16T19:37:36.143 INFO:teuthology.orchestra.run.smithi118.stdout: debug asserts on shutdown = true 2021-09-16T19:37:36.144 INFO:teuthology.orchestra.run.smithi118.stdout: mon health detail to clog = false 2021-09-16T19:37:36.144 INFO:teuthology.orchestra.run.smithi118.stdout: mon host = 172.21.15.118 2021-09-16T19:37:36.144 INFO:teuthology.orchestra.run.smithi118.stdout: fsid = 64e7c9b4-a739-4726-94e9-afe7eeb05b8f 2021-09-16T19:37:36.144 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.144 INFO:teuthology.orchestra.run.smithi118.stdout:[osd] 2021-09-16T19:37:36.145 INFO:teuthology.orchestra.run.smithi118.stdout: osd journal size = 100 2021-09-16T19:37:36.145 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.145 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub load threshold = 5.0 2021-09-16T19:37:36.145 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub max interval = 600 2021-09-16T19:37:36.146 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.146 INFO:teuthology.orchestra.run.smithi118.stdout: osd recover clone overlap = true 2021-09-16T19:37:36.147 INFO:teuthology.orchestra.run.smithi118.stdout: osd recovery max chunk = 1048576 2021-09-16T19:37:36.147 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.147 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug shutdown = true 2021-09-16T19:37:36.147 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug op order = true 2021-09-16T19:37:36.147 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify stray on activate = true 2021-09-16T19:37:36.148 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.148 INFO:teuthology.orchestra.run.smithi118.stdout: osd open classes on start = true 2021-09-16T19:37:36.148 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug pg log writeout = true 2021-09-16T19:37:36.148 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.149 INFO:teuthology.orchestra.run.smithi118.stdout: osd deep scrub update digest min age = 30 2021-09-16T19:37:36.149 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.149 INFO:teuthology.orchestra.run.smithi118.stdout: osd map max advance = 10 2021-09-16T19:37:36.149 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.149 INFO:teuthology.orchestra.run.smithi118.stdout: journal zero on create = true 2021-09-16T19:37:36.150 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.150 INFO:teuthology.orchestra.run.smithi118.stdout: filestore ondisk finisher threads = 3 2021-09-16T19:37:36.150 INFO:teuthology.orchestra.run.smithi118.stdout: filestore apply finisher threads = 3 2021-09-16T19:37:36.150 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.151 INFO:teuthology.orchestra.run.smithi118.stdout: bdev debug aio = true 2021-09-16T19:37:36.151 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug misdirected ops = true 2021-09-16T19:37:36.151 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2021-09-16T19:37:36.151 INFO:teuthology.orchestra.run.smithi118.stdout: debug osd = 20 2021-09-16T19:37:36.151 INFO:teuthology.orchestra.run.smithi118.stdout: rocksdb delete range threshold = 0 2021-09-16T19:37:36.152 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.152 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr] 2021-09-16T19:37:36.152 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2021-09-16T19:37:36.152 INFO:teuthology.orchestra.run.smithi118.stdout: debug mgr = 20 2021-09-16T19:37:36.152 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2021-09-16T19:37:36.153 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2021-09-16T19:37:36.153 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2021-09-16T19:37:36.153 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2021-09-16T19:37:36.153 INFO:teuthology.orchestra.run.smithi118.stdout: mgr/telemetry/nag = false 2021-09-16T19:37:36.153 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.154 INFO:teuthology.orchestra.run.smithi118.stdout:[mon] 2021-09-16T19:37:36.154 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2021-09-16T19:37:36.154 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2021-09-16T19:37:36.154 INFO:teuthology.orchestra.run.smithi118.stdout: debug paxos = 20 2021-09-16T19:37:36.155 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2021-09-16T19:37:36.155 INFO:teuthology.orchestra.run.smithi118.stdout: mon data avail warn = 5 2021-09-16T19:37:36.155 INFO:teuthology.orchestra.run.smithi118.stdout: mon mgr mkfs grace = 240 2021-09-16T19:37:36.155 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2021-09-16T19:37:36.155 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd reporter subtree level = osd 2021-09-16T19:37:36.156 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd prime pg temp = true 2021-09-16T19:37:36.156 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2021-09-16T19:37:36.156 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.156 INFO:teuthology.orchestra.run.smithi118.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-16T19:37:36.157 INFO:teuthology.orchestra.run.smithi118.stdout: auth mon ticket ttl = 660 # 11m 2021-09-16T19:37:36.157 INFO:teuthology.orchestra.run.smithi118.stdout: auth service ticket ttl = 240 # 4m 2021-09-16T19:37:36.157 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.157 INFO:teuthology.orchestra.run.smithi118.stdout: # don't complain about insecure global_id in the test suite 2021-09-16T19:37:36.157 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-16T19:37:36.159 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-16T19:37:36.159 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:36.159 INFO:teuthology.orchestra.run.smithi118.stdout:[client] 2021-09-16T19:37:36.159 INFO:teuthology.orchestra.run.smithi118.stdout: rgw cache enabled = true 2021-09-16T19:37:36.160 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable ops log = true 2021-09-16T19:37:36.162 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable usage log = true 2021-09-16T19:37:36.162 INFO:teuthology.orchestra.run.smithi118.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-16T19:37:36.162 INFO:teuthology.orchestra.run.smithi118.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-16T19:37:36.163 INFO:teuthology.orchestra.run.smithi118.stdout: debug rgw = 20 2021-09-16T19:37:36.163 INFO:teuthology.orchestra.run.smithi118.stdout: rgw crypt require ssl = False 2021-09-16T19:37:36.163 INFO:teuthology.orchestra.run.smithi118.stdout: rgw crypt s3 kms backend = testing 2021-09-16T19:37:36.163 INFO:teuthology.orchestra.run.smithi118.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-16T19:37:36.164 INFO:teuthology.orchestra.run.smithi118.stdout: rgw curl low speed time = 300 2021-09-16T19:37:36.164 INFO:teuthology.orchestra.run.smithi118.stdout: rgw data log num shards = 4 2021-09-16T19:37:36.164 INFO:teuthology.orchestra.run.smithi118.stdout: rgw md log max shards = 4 2021-09-16T19:37:36.164 INFO:teuthology.orchestra.run.smithi118.stdout: rgw sync log trim interval = 0 2021-09-16T19:37:36.164 INFO:teuthology.orchestra.run.smithi118.stdout: rgw sync meta inject err probability = 0.1 2021-09-16T19:37:36.165 INFO:teuthology.orchestra.run.smithi118.stdout: rgw sync obj etag verify = True 2021-09-16T19:37:36.165 INFO:teuthology.orchestra.run.smithi118.stdout: setgroup = ceph 2021-09-16T19:37:36.165 INFO:teuthology.orchestra.run.smithi118.stdout: setuser = ceph 2021-09-16T19:37:36.165 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.a] 2021-09-16T19:37:36.165 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.0] 2021-09-16T19:37:36.166 INFO:teuthology.orchestra.run.smithi118.stdout: osd_max_omap_entries_per_request = 10 2021-09-16T19:37:36.166 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.1] 2021-09-16T19:37:36.166 INFO:teuthology.orchestra.run.smithi118.stdout: osd_max_omap_entries_per_request = 1000 2021-09-16T19:37:36.166 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.2] 2021-09-16T19:37:36.167 INFO:teuthology.orchestra.run.smithi118.stdout: osd_max_omap_entries_per_request = 10000 2021-09-16T19:37:36.168 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2021-09-16T19:37:36.168 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2021-09-16T19:37:36.258 INFO:tasks.ceph:Copying monmap to all nodes... 2021-09-16T19:37:36.259 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:36.260 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2021-09-16T19:37:36.279 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:36.280 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2021-09-16T19:37:36.338 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi109.front.sepia.ceph.com 2021-09-16T19:37:36.339 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:36.339 DEBUG:teuthology.orchestra.run.smithi109:> sudo dd of=/etc/ceph/c2.keyring 2021-09-16T19:37:36.339 DEBUG:teuthology.orchestra.run.smithi109:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-16T19:37:36.379 DEBUG:teuthology.orchestra.run.smithi109:> set -ex 2021-09-16T19:37:36.380 DEBUG:teuthology.orchestra.run.smithi109:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-09-16T19:37:36.441 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi118.front.sepia.ceph.com 2021-09-16T19:37:36.442 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:36.442 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/c2.keyring 2021-09-16T19:37:36.443 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-16T19:37:36.481 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:36.481 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-09-16T19:37:36.538 INFO:tasks.ceph:Setting up mon nodes... 2021-09-16T19:37:36.539 INFO:tasks.ceph:Setting up mgr nodes... 2021-09-16T19:37:36.539 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2021-09-16T19:37:36.627 INFO:teuthology.orchestra.run.smithi118.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2021-09-16T19:37:36.629 INFO:tasks.ceph:Setting up mds nodes... 2021-09-16T19:37:36.629 INFO:tasks.ceph_client:Setting up client nodes... 2021-09-16T19:37:36.630 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2021-09-16T19:37:36.677 INFO:teuthology.orchestra.run.smithi118.stdout:creating /etc/ceph/c2.client.0.keyring 2021-09-16T19:37:36.689 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2021-09-16T19:37:36.767 INFO:teuthology.orchestra.run.smithi118.stdout:creating /etc/ceph/c2.client.1.keyring 2021-09-16T19:37:36.779 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-09-16T19:37:36.781 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi109.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-16T19:37:36.781 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2021-09-16T19:37:36.845 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-16T19:37:36.845 INFO:tasks.ceph:role: c2.osd.0 2021-09-16T19:37:36.845 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com 2021-09-16T19:37:36.846 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-09-16T19:37:36.909 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-09-16T19:37:36.910 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-16T19:37:36.910 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-16T19:37:36.910 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 2021-09-16T19:37:36.911 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-16T19:37:36.911 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2021-09-16T19:37:36.911 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-16T19:37:36.911 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-16T19:37:36.912 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-16T19:37:36.912 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-16T19:37:36.932 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2021-09-16T19:37:36.934 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2021-09-16T19:37:36.934 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2021-09-16T19:37:37.012 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2021-09-16T19:37:37.081 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2021-09-16T19:37:37.145 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-16T19:37:37.145 INFO:tasks.ceph:role: c2.osd.1 2021-09-16T19:37:37.146 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com 2021-09-16T19:37:37.146 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-09-16T19:37:37.208 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-09-16T19:37:37.209 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-16T19:37:37.209 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-16T19:37:37.209 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 2021-09-16T19:37:37.209 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-16T19:37:37.210 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2021-09-16T19:37:37.210 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-16T19:37:37.210 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-16T19:37:37.210 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-16T19:37:37.211 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-16T19:37:37.231 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2021-09-16T19:37:37.232 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2021-09-16T19:37:37.233 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2021-09-16T19:37:37.308 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2021-09-16T19:37:37.375 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2021-09-16T19:37:37.439 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-16T19:37:37.439 INFO:tasks.ceph:role: c2.osd.2 2021-09-16T19:37:37.440 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com 2021-09-16T19:37:37.440 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-09-16T19:37:37.502 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-09-16T19:37:37.503 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-16T19:37:37.503 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-16T19:37:37.503 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 2021-09-16T19:37:37.503 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-16T19:37:37.504 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2021-09-16T19:37:37.504 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-16T19:37:37.504 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-16T19:37:37.505 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-16T19:37:37.505 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-16T19:37:37.525 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2021-09-16T19:37:37.526 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2021-09-16T19:37:37.527 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2021-09-16T19:37:37.602 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2021-09-16T19:37:37.669 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-16T19:37:37.750 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:37.748+0000 7f07ed9490c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2021-09-16T19:37:37.750 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:37.748+0000 7f07ed9490c0 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2021-09-16T19:37:37.751 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:37.748+0000 7f07ed9490c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-16T19:37:37.751 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:37.748+0000 7f07ed9490c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-16T19:37:37.751 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:37.748+0000 7f07ed9490c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-16T19:37:37.751 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:37.748+0000 7f07ed9490c0 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2021-09-16T19:37:39.830 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2021-09-16T19:37:39.900 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-16T19:37:39.988 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:39.986+0000 7ff4f0ca70c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2021-09-16T19:37:39.989 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:39.986+0000 7ff4f0ca70c0 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2021-09-16T19:37:39.989 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:39.986+0000 7ff4f0ca70c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-16T19:37:39.989 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:39.986+0000 7ff4f0ca70c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-16T19:37:39.989 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:39.986+0000 7ff4f0ca70c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-16T19:37:39.990 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:39.987+0000 7ff4f0ca70c0 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2021-09-16T19:37:42.069 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2021-09-16T19:37:42.098 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-16T19:37:42.186 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:42.184+0000 7f3b2a8140c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2021-09-16T19:37:42.187 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:42.185+0000 7f3b2a8140c0 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2021-09-16T19:37:42.187 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:42.185+0000 7f3b2a8140c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-16T19:37:42.187 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:42.185+0000 7f3b2a8140c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-16T19:37:42.188 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:42.185+0000 7f3b2a8140c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-16T19:37:42.188 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:37:42.185+0000 7f3b2a8140c0 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2021-09-16T19:37:44.278 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2021-09-16T19:37:44.307 INFO:tasks.ceph:Reading keys from all nodes... 2021-09-16T19:37:44.307 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:44.308 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2021-09-16T19:37:44.375 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:44.376 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2021-09-16T19:37:44.443 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:44.444 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2021-09-16T19:37:44.512 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:44.512 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2021-09-16T19:37:44.580 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:44.585 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2021-09-16T19:37:44.636 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:44.637 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2021-09-16T19:37:44.692 INFO:tasks.ceph:Adding keys to all mons... 2021-09-16T19:37:44.692 DEBUG:teuthology.orchestra.run.smithi118:> sudo tee -a /etc/ceph/c2.keyring 2021-09-16T19:37:44.757 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr.x] 2021-09-16T19:37:44.758 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAAnUNhm75fJRAAxWk1kZWKaqUMyeDUdzZoFw== 2021-09-16T19:37:44.758 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.0] 2021-09-16T19:37:44.758 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQABnUNhZHqrLBAAMZyw9Ai1r5z8JBU+PiN52g== 2021-09-16T19:37:44.759 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.1] 2021-09-16T19:37:44.759 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQADnUNhpavhOhAAEqaE9N6BrIp6tjD0DRUhdQ== 2021-09-16T19:37:44.759 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.2] 2021-09-16T19:37:44.759 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAGnUNhwWgYCxAAeebnuDgzCJp0mtDKstVU2A== 2021-09-16T19:37:44.760 INFO:teuthology.orchestra.run.smithi118.stdout:[client.0] 2021-09-16T19:37:44.760 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAAnUNhRS5dKBAARvEnAqQwVpwSLMRsmbYpHA== 2021-09-16T19:37:44.760 INFO:teuthology.orchestra.run.smithi118.stdout:[client.1] 2021-09-16T19:37:44.760 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAAnUNh2hO7LRAAuXEEPFGpFrgZQn0Om3gusQ== 2021-09-16T19:37:44.762 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-09-16T19:37:44.852 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-16T19:37:44.899 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-16T19:37:44.945 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2021-09-16T19:37:44.991 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-16T19:37:45.041 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-16T19:37:45.090 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-09-16T19:37:45.091 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2021-09-16T19:37:45.115 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2021-09-16T19:37:45.205 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2021-09-16T19:37:45.229 DEBUG:teuthology.orchestra.run.smithi118:> rm -- /home/ubuntu/cephtest/c2.monmap 2021-09-16T19:37:45.284 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2021-09-16T19:37:45.284 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2021-09-16T19:37:45.285 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2021-09-16T19:37:45.326 INFO:tasks.ceph.c2.mon.a:Started 2021-09-16T19:37:45.327 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2021-09-16T19:37:45.328 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2021-09-16T19:37:45.328 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2021-09-16T19:37:45.330 INFO:tasks.ceph.c2.mgr.x:Started 2021-09-16T19:37:45.331 INFO:tasks.ceph:Setting crush tunables to default 2021-09-16T19:37:45.331 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster c2 osd crush tunables default 2021-09-16T19:37:45.525 INFO:teuthology.orchestra.run.smithi118.stderr:adjusted tunables profile to default 2021-09-16T19:37:45.535 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2021-09-16T19:37:45.535 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:45.537 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2021-09-16T19:37:45.560 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:45.561 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2021-09-16T19:37:45.625 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2021-09-16T19:37:45.625 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2021-09-16T19:37:45.690 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster c2 osd new 596646bd-383b-44d9-b45f-4e95f9954341 0 2021-09-16T19:37:45.932 INFO:teuthology.orchestra.run.smithi118.stdout:0 2021-09-16T19:37:45.945 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster c2 osd new 7a94fa50-df79-4a99-a3c3-065bd79f59bf 1 2021-09-16T19:37:46.146 INFO:teuthology.orchestra.run.smithi118.stdout:1 2021-09-16T19:37:46.159 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster c2 osd new da00bc5c-13b2-4799-bc81-30cca9ed01c4 2 2021-09-16T19:37:46.361 INFO:teuthology.orchestra.run.smithi118.stdout:2 2021-09-16T19:37:46.374 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2021-09-16T19:37:46.375 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2021-09-16T19:37:46.378 INFO:tasks.ceph.c2.osd.0:Started 2021-09-16T19:37:46.378 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2021-09-16T19:37:46.379 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2021-09-16T19:37:46.381 INFO:tasks.ceph.c2.osd.1:Started 2021-09-16T19:37:46.381 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2021-09-16T19:37:46.381 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2021-09-16T19:37:46.383 INFO:tasks.ceph.c2.osd.2:Started 2021-09-16T19:37:46.384 INFO:tasks.ceph:Waiting for OSDs to come up 2021-09-16T19:37:46.687 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-09-16T19:37:46.866 INFO:teuthology.misc.health.smithi118.stdout: 2021-09-16T19:37:46.866 INFO:teuthology.misc.health.smithi118.stdout:{"epoch":5,"fsid":"64e7c9b4-a739-4726-94e9-afe7eeb05b8f","created":"2021-09-16T19:37:45.420375+0000","modified":"2021-09-16T19:37:46.359738+0000","last_up_change":"0.000000","last_in_change":"2021-09-16T19:37:46.359738+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"596646bd-383b-44d9-b45f-4e95f9954341","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7a94fa50-df79-4a99-a3c3-065bd79f59bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"da00bc5c-13b2-4799-bc81-30cca9ed01c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-16T19:37:46.877 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-09-16T19:37:46.971 INFO:tasks.ceph.c2.osd.2.smithi118.stderr:2021-09-16T19:37:46.969+0000 7f4373a920c0 -1 Falling back to public interface 2021-09-16T19:37:46.972 INFO:tasks.ceph.c2.osd.0.smithi118.stderr:2021-09-16T19:37:46.969+0000 7f67430ca0c0 -1 Falling back to public interface 2021-09-16T19:37:46.972 INFO:tasks.ceph.c2.osd.1.smithi118.stderr:2021-09-16T19:37:46.969+0000 7fad319100c0 -1 Falling back to public interface 2021-09-16T19:37:46.973 INFO:tasks.ceph.c2.osd.0.smithi118.stderr:2021-09-16T19:37:46.970+0000 7f673d28f700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6804/0: (98) Address already in use 2021-09-16T19:37:46.974 INFO:tasks.ceph.c2.osd.0.smithi118.stderr:2021-09-16T19:37:46.970+0000 7f673d28f700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6805/0: (98) Address already in use 2021-09-16T19:37:46.975 INFO:tasks.ceph.c2.osd.2.smithi118.stderr:2021-09-16T19:37:46.970+0000 7f436dc57700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6808/0: (98) Address already in use 2021-09-16T19:37:46.975 INFO:tasks.ceph.c2.osd.2.smithi118.stderr:2021-09-16T19:37:46.970+0000 7f436dc57700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6809/0: (98) Address already in use 2021-09-16T19:37:49.702 INFO:tasks.ceph.c2.osd.0.smithi118.stderr:2021-09-16T19:37:49.702+0000 7f67430ca0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-09-16T19:37:49.710 INFO:tasks.ceph.c2.osd.1.smithi118.stderr:2021-09-16T19:37:49.710+0000 7fad319100c0 -1 osd.1 0 log_to_monitors {default=true} 2021-09-16T19:37:49.715 INFO:tasks.ceph.c2.osd.2.smithi118.stderr:2021-09-16T19:37:49.715+0000 7f4373a920c0 -1 osd.2 0 log_to_monitors {default=true} 2021-09-16T19:37:51.469 INFO:tasks.ceph.c2.osd.1.smithi118.stderr:2021-09-16T19:37:51.468+0000 7fad282ce700 -1 osd.1 0 waiting for initial osdmap 2021-09-16T19:37:51.470 INFO:tasks.ceph.c2.osd.1.smithi118.stderr:2021-09-16T19:37:51.470+0000 7fad20c61700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-16T19:37:51.547 INFO:tasks.ceph.c2.osd.2.smithi118.stderr:2021-09-16T19:37:51.547+0000 7f436a450700 -1 osd.2 0 waiting for initial osdmap 2021-09-16T19:37:51.548 INFO:tasks.ceph.c2.osd.2.smithi118.stderr:2021-09-16T19:37:51.548+0000 7f4362de3700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-16T19:37:51.882 INFO:tasks.ceph.c2.osd.0.smithi118.stderr:2021-09-16T19:37:51.881+0000 7f6739a88700 -1 osd.0 0 waiting for initial osdmap 2021-09-16T19:37:51.883 INFO:tasks.ceph.c2.osd.0.smithi118.stderr:2021-09-16T19:37:51.883+0000 7f673241b700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-16T19:37:53.181 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-09-16T19:37:53.465 INFO:teuthology.misc.health.smithi118.stdout: 2021-09-16T19:37:53.465 INFO:teuthology.misc.health.smithi118.stdout:{"epoch":8,"fsid":"64e7c9b4-a739-4726-94e9-afe7eeb05b8f","created":"2021-09-16T19:37:45.420375+0000","modified":"2021-09-16T19:37:52.466497+0000","last_up_change":"2021-09-16T19:37:52.466497+0000","last_in_change":"2021-09-16T19:37:46.359738+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"596646bd-383b-44d9-b45f-4e95f9954341","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":31550},{"type":"v1","addr":"172.21.15.118:6808","nonce":31550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":31550},{"type":"v1","addr":"172.21.15.118:6814","nonce":31550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":31550},{"type":"v1","addr":"172.21.15.118:6823","nonce":31550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":31550},{"type":"v1","addr":"172.21.15.118:6819","nonce":31550}]},"public_addr":"172.21.15.118:6808/31550","cluster_addr":"172.21.15.118:6814/31550","heartbeat_back_addr":"172.21.15.118:6823/31550","heartbeat_front_addr":"172.21.15.118:6819/31550","state":["exists","up"]},{"osd":1,"uuid":"7a94fa50-df79-4a99-a3c3-065bd79f59bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":31551},{"type":"v1","addr":"172.21.15.118:6807","nonce":31551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":31551},{"type":"v1","addr":"172.21.15.118:6812","nonce":31551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":31551},{"type":"v1","addr":"172.21.15.118:6822","nonce":31551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6815","nonce":31551},{"type":"v1","addr":"172.21.15.118:6818","nonce":31551}]},"public_addr":"172.21.15.118:6807/31551","cluster_addr":"172.21.15.118:6812/31551","heartbeat_back_addr":"172.21.15.118:6822/31551","heartbeat_front_addr":"172.21.15.118:6818/31551","state":["exists","up"]},{"osd":2,"uuid":"da00bc5c-13b2-4799-bc81-30cca9ed01c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":31552},{"type":"v1","addr":"172.21.15.118:6801","nonce":31552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":31552},{"type":"v1","addr":"172.21.15.118:6803","nonce":31552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":31552},{"type":"v1","addr":"172.21.15.118:6817","nonce":31552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":31552},{"type":"v1","addr":"172.21.15.118:6810","nonce":31552}]},"public_addr":"172.21.15.118:6801/31552","cluster_addr":"172.21.15.118:6803/31552","heartbeat_back_addr":"172.21.15.118:6817/31552","heartbeat_front_addr":"172.21.15.118:6810/31552","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-16T19:37:53.478 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-09-16T19:37:53.479 INFO:tasks.ceph:Creating RBD pool 2021-09-16T19:37:53.479 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster c2 osd pool create rbd 8 2021-09-16T19:37:54.475 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'rbd' created 2021-09-16T19:37:54.488 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-09-16T19:37:55.478 INFO:teuthology.orchestra.run.smithi118.stderr:enabled application 'rbd' on pool 'rbd' 2021-09-16T19:37:55.492 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2021-09-16T19:37:55.493 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-16T19:37:55.494 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-09-16T19:37:55.857 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:55.857 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":11,"fsid":"64e7c9b4-a739-4726-94e9-afe7eeb05b8f","created":"2021-09-16T19:37:45.420375+0000","modified":"2021-09-16T19:37:55.468664+0000","last_up_change":"2021-09-16T19:37:52.466497+0000","last_in_change":"2021-09-16T19:37:46.359738+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-16T19:37:53.776068+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-16T19:37:54.286530+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"596646bd-383b-44d9-b45f-4e95f9954341","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":31550},{"type":"v1","addr":"172.21.15.118:6808","nonce":31550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":31550},{"type":"v1","addr":"172.21.15.118:6814","nonce":31550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":31550},{"type":"v1","addr":"172.21.15.118:6823","nonce":31550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":31550},{"type":"v1","addr":"172.21.15.118:6819","nonce":31550}]},"public_addr":"172.21.15.118:6808/31550","cluster_addr":"172.21.15.118:6814/31550","heartbeat_back_addr":"172.21.15.118:6823/31550","heartbeat_front_addr":"172.21.15.118:6819/31550","state":["exists","up"]},{"osd":1,"uuid":"7a94fa50-df79-4a99-a3c3-065bd79f59bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":31551},{"type":"v1","addr":"172.21.15.118:6807","nonce":31551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":31551},{"type":"v1","addr":"172.21.15.118:6812","nonce":31551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":31551},{"type":"v1","addr":"172.21.15.118:6822","nonce":31551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6815","nonce":31551},{"type":"v1","addr":"172.21.15.118:6818","nonce":31551}]},"public_addr":"172.21.15.118:6807/31551","cluster_addr":"172.21.15.118:6812/31551","heartbeat_back_addr":"172.21.15.118:6822/31551","heartbeat_front_addr":"172.21.15.118:6818/31551","state":["exists","up"]},{"osd":2,"uuid":"da00bc5c-13b2-4799-bc81-30cca9ed01c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":31552},{"type":"v1","addr":"172.21.15.118:6801","nonce":31552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":31552},{"type":"v1","addr":"172.21.15.118:6803","nonce":31552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":31552},{"type":"v1","addr":"172.21.15.118:6817","nonce":31552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":31552},{"type":"v1","addr":"172.21.15.118:6810","nonce":31552}]},"public_addr":"172.21.15.118:6801/31552","cluster_addr":"172.21.15.118:6803/31552","heartbeat_back_addr":"172.21.15.118:6817/31552","heartbeat_front_addr":"172.21.15.118:6810/31552","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:50.722823+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:50.720292+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:50.748183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-16T19:37:55.872 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-09-16T19:37:53.776068+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2021-09-16T19:37:54.286530+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2021-09-16T19:37:55.873 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get rbd pg_num 2021-09-16T19:37:56.159 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 8 2021-09-16T19:37:56.172 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get .mgr pg_num 2021-09-16T19:37:56.457 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 1 2021-09-16T19:37:56.471 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2021-09-16T19:37:56.471 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2021-09-16T19:37:56.472 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2021-09-16T19:37:56.803 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:56.813 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":31231},{"type":"v1","addr":"172.21.15.118:6825","nonce":31231}]},"active_addr":"172.21.15.118:6825/31231","active_change":"2021-09-16T19:37:50.255055+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"Determines the behavior of the autoscaler algorithm `scale-up` means that it starts out with minmum pgs and scales up when there is pressure, `scale-down` means starts out with full pgs and scales down when there is pressure ","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share perf counter metrics summed across the whole cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":132865286}]},{"addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":3116073605}]},{"addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":3869402956}]},{"addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":1596565366}]}]}} 2021-09-16T19:37:56.816 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2021-09-16T19:37:56.816 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2021-09-16T19:37:56.816 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-16T19:37:57.101 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:57.101 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":12,"fsid":"64e7c9b4-a739-4726-94e9-afe7eeb05b8f","created":"2021-09-16T19:37:45.420375+0000","modified":"2021-09-16T19:37:56.469690+0000","last_up_change":"2021-09-16T19:37:52.466497+0000","last_in_change":"2021-09-16T19:37:46.359738+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-16T19:37:53.776068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-16T19:37:54.286530+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"596646bd-383b-44d9-b45f-4e95f9954341","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":31550},{"type":"v1","addr":"172.21.15.118:6808","nonce":31550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":31550},{"type":"v1","addr":"172.21.15.118:6814","nonce":31550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":31550},{"type":"v1","addr":"172.21.15.118:6823","nonce":31550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":31550},{"type":"v1","addr":"172.21.15.118:6819","nonce":31550}]},"public_addr":"172.21.15.118:6808/31550","cluster_addr":"172.21.15.118:6814/31550","heartbeat_back_addr":"172.21.15.118:6823/31550","heartbeat_front_addr":"172.21.15.118:6819/31550","state":["exists","up"]},{"osd":1,"uuid":"7a94fa50-df79-4a99-a3c3-065bd79f59bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":31551},{"type":"v1","addr":"172.21.15.118:6807","nonce":31551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":31551},{"type":"v1","addr":"172.21.15.118:6812","nonce":31551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":31551},{"type":"v1","addr":"172.21.15.118:6822","nonce":31551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6815","nonce":31551},{"type":"v1","addr":"172.21.15.118:6818","nonce":31551}]},"public_addr":"172.21.15.118:6807/31551","cluster_addr":"172.21.15.118:6812/31551","heartbeat_back_addr":"172.21.15.118:6822/31551","heartbeat_front_addr":"172.21.15.118:6818/31551","state":["exists","up"]},{"osd":2,"uuid":"da00bc5c-13b2-4799-bc81-30cca9ed01c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":31552},{"type":"v1","addr":"172.21.15.118:6801","nonce":31552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":31552},{"type":"v1","addr":"172.21.15.118:6803","nonce":31552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":31552},{"type":"v1","addr":"172.21.15.118:6817","nonce":31552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":31552},{"type":"v1","addr":"172.21.15.118:6810","nonce":31552}]},"public_addr":"172.21.15.118:6801/31552","cluster_addr":"172.21.15.118:6803/31552","heartbeat_back_addr":"172.21.15.118:6817/31552","heartbeat_front_addr":"172.21.15.118:6810/31552","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:50.722823+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:50.720292+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:50.748183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-16T19:37:57.114 INFO:tasks.ceph.ceph_manager.c2:all up! 2021-09-16T19:37:57.114 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-16T19:37:57.398 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:37:57.399 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":12,"fsid":"64e7c9b4-a739-4726-94e9-afe7eeb05b8f","created":"2021-09-16T19:37:45.420375+0000","modified":"2021-09-16T19:37:56.469690+0000","last_up_change":"2021-09-16T19:37:52.466497+0000","last_in_change":"2021-09-16T19:37:46.359738+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-16T19:37:53.776068+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-16T19:37:54.286530+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"596646bd-383b-44d9-b45f-4e95f9954341","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":31550},{"type":"v1","addr":"172.21.15.118:6808","nonce":31550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":31550},{"type":"v1","addr":"172.21.15.118:6814","nonce":31550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6821","nonce":31550},{"type":"v1","addr":"172.21.15.118:6823","nonce":31550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":31550},{"type":"v1","addr":"172.21.15.118:6819","nonce":31550}]},"public_addr":"172.21.15.118:6808/31550","cluster_addr":"172.21.15.118:6814/31550","heartbeat_back_addr":"172.21.15.118:6823/31550","heartbeat_front_addr":"172.21.15.118:6819/31550","state":["exists","up"]},{"osd":1,"uuid":"7a94fa50-df79-4a99-a3c3-065bd79f59bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":31551},{"type":"v1","addr":"172.21.15.118:6807","nonce":31551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":31551},{"type":"v1","addr":"172.21.15.118:6812","nonce":31551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":31551},{"type":"v1","addr":"172.21.15.118:6822","nonce":31551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6815","nonce":31551},{"type":"v1","addr":"172.21.15.118:6818","nonce":31551}]},"public_addr":"172.21.15.118:6807/31551","cluster_addr":"172.21.15.118:6812/31551","heartbeat_back_addr":"172.21.15.118:6822/31551","heartbeat_front_addr":"172.21.15.118:6818/31551","state":["exists","up"]},{"osd":2,"uuid":"da00bc5c-13b2-4799-bc81-30cca9ed01c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":31552},{"type":"v1","addr":"172.21.15.118:6801","nonce":31552}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":31552},{"type":"v1","addr":"172.21.15.118:6803","nonce":31552}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":31552},{"type":"v1","addr":"172.21.15.118:6817","nonce":31552}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":31552},{"type":"v1","addr":"172.21.15.118:6810","nonce":31552}]},"public_addr":"172.21.15.118:6801/31552","cluster_addr":"172.21.15.118:6803/31552","heartbeat_back_addr":"172.21.15.118:6817/31552","heartbeat_front_addr":"172.21.15.118:6810/31552","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:50.722823+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:50.720292+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-16T19:37:50.748183+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-16T19:37:57.412 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2021-09-16T19:37:57.555 INFO:teuthology.orchestra.run.smithi118.stdout:34359738371 2021-09-16T19:37:57.556 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2021-09-16T19:37:57.699 INFO:teuthology.orchestra.run.smithi118.stdout:34359738372 2021-09-16T19:37:57.699 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2021-09-16T19:37:57.842 INFO:teuthology.orchestra.run.smithi118.stdout:34359738371 2021-09-16T19:37:57.843 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-09-16T19:37:58.129 INFO:teuthology.orchestra.run.smithi118.stdout:34359738370 2021-09-16T19:37:58.142 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2021-09-16T19:37:59.142 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-09-16T19:37:59.446 INFO:teuthology.orchestra.run.smithi118.stdout:34359738371 2021-09-16T19:37:59.459 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.0 2021-09-16T19:37:59.460 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2021-09-16T19:37:59.741 INFO:teuthology.orchestra.run.smithi118.stdout:34359738372 2021-09-16T19:37:59.754 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.1 2021-09-16T19:37:59.755 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2021-09-16T19:38:00.035 INFO:teuthology.orchestra.run.smithi118.stdout:34359738371 2021-09-16T19:38:00.047 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.2 2021-09-16T19:38:00.048 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2021-09-16T19:38:00.048 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-09-16T19:38:00.330 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:38:00.332 INFO:teuthology.orchestra.run.smithi118.stderr:dumped all 2021-09-16T19:38:00.344 INFO:teuthology.orchestra.run.smithi118.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2021-09-16T19:38:00.259549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24968,"kb_used_data":1456,"kb_used_omap":0,"kb_used_meta":23232,"kb_avail":314547832,"statfs":{"total":322122547200,"available":322096979968,"internally_reserved":0,"allocated":1490944,"data_stored":1258871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23789568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000325"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.477075+0000","last_change":"2021-09-16T19:37:55.477075+0000","last_active":"2021-09-16T19:37:55.477075+0000","last_peered":"2021-09-16T19:37:55.477075+0000","last_clean":"2021-09-16T19:37:55.477075+0000","last_became_active":"2021-09-16T19:37:55.476645+0000","last_became_peered":"2021-09-16T19:37:55.476645+0000","last_unstale":"2021-09-16T19:37:55.477075+0000","last_undegraded":"2021-09-16T19:37:55.477075+0000","last_fullsized":"2021-09-16T19:37:55.477075+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.473509+0000","last_change":"2021-09-16T19:37:55.473509+0000","last_active":"2021-09-16T19:37:55.473509+0000","last_peered":"2021-09-16T19:37:55.473509+0000","last_clean":"2021-09-16T19:37:55.473509+0000","last_became_active":"2021-09-16T19:37:55.473257+0000","last_became_peered":"2021-09-16T19:37:55.473257+0000","last_unstale":"2021-09-16T19:37:55.473509+0000","last_undegraded":"2021-09-16T19:37:55.473509+0000","last_fullsized":"2021-09-16T19:37:55.473509+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.477326+0000","last_change":"2021-09-16T19:37:55.475859+0000","last_active":"2021-09-16T19:37:55.477326+0000","last_peered":"2021-09-16T19:37:55.477326+0000","last_clean":"2021-09-16T19:37:55.477326+0000","last_became_active":"2021-09-16T19:37:55.475599+0000","last_became_peered":"2021-09-16T19:37:55.475599+0000","last_unstale":"2021-09-16T19:37:55.477326+0000","last_undegraded":"2021-09-16T19:37:55.477326+0000","last_fullsized":"2021-09-16T19:37:55.477326+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.475059+0000","last_change":"2021-09-16T19:37:55.475059+0000","last_active":"2021-09-16T19:37:55.475059+0000","last_peered":"2021-09-16T19:37:55.475059+0000","last_clean":"2021-09-16T19:37:55.475059+0000","last_became_active":"2021-09-16T19:37:55.474791+0000","last_became_peered":"2021-09-16T19:37:55.474791+0000","last_unstale":"2021-09-16T19:37:55.475059+0000","last_undegraded":"2021-09-16T19:37:55.475059+0000","last_fullsized":"2021-09-16T19:37:55.475059+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.474282+0000","last_change":"2021-09-16T19:37:55.474282+0000","last_active":"2021-09-16T19:37:55.474282+0000","last_peered":"2021-09-16T19:37:55.474282+0000","last_clean":"2021-09-16T19:37:55.474282+0000","last_became_active":"2021-09-16T19:37:55.474038+0000","last_became_peered":"2021-09-16T19:37:55.474038+0000","last_unstale":"2021-09-16T19:37:55.474282+0000","last_undegraded":"2021-09-16T19:37:55.474282+0000","last_fullsized":"2021-09-16T19:37:55.474282+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.867873+0000","last_change":"2021-09-16T19:37:55.477001+0000","last_active":"2021-09-16T19:37:55.867873+0000","last_peered":"2021-09-16T19:37:55.867873+0000","last_clean":"2021-09-16T19:37:55.867873+0000","last_became_active":"2021-09-16T19:37:55.476216+0000","last_became_peered":"2021-09-16T19:37:55.476216+0000","last_unstale":"2021-09-16T19:37:55.867873+0000","last_undegraded":"2021-09-16T19:37:55.867873+0000","last_fullsized":"2021-09-16T19:37:55.867873+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.476649+0000","last_change":"2021-09-16T19:37:55.476649+0000","last_active":"2021-09-16T19:37:55.476649+0000","last_peered":"2021-09-16T19:37:55.476649+0000","last_clean":"2021-09-16T19:37:55.476649+0000","last_became_active":"2021-09-16T19:37:55.476390+0000","last_became_peered":"2021-09-16T19:37:55.476390+0000","last_unstale":"2021-09-16T19:37:55.476649+0000","last_undegraded":"2021-09-16T19:37:55.476649+0000","last_fullsized":"2021-09-16T19:37:55.476649+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.475889+0000","last_change":"2021-09-16T19:37:55.475889+0000","last_active":"2021-09-16T19:37:55.475889+0000","last_peered":"2021-09-16T19:37:55.475889+0000","last_clean":"2021-09-16T19:37:55.475889+0000","last_became_active":"2021-09-16T19:37:55.475637+0000","last_became_peered":"2021-09-16T19:37:55.475637+0000","last_unstale":"2021-09-16T19:37:55.475889+0000","last_undegraded":"2021-09-16T19:37:55.475889+0000","last_fullsized":"2021-09-16T19:37:55.475889+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.477320+0000","last_change":"2021-09-16T19:37:55.475884+0000","last_active":"2021-09-16T19:37:55.477320+0000","last_peered":"2021-09-16T19:37:55.477320+0000","last_clean":"2021-09-16T19:37:55.477320+0000","last_became_active":"2021-09-16T19:37:55.475637+0000","last_became_peered":"2021-09-16T19:37:55.475637+0000","last_unstale":"2021-09-16T19:37:55.477320+0000","last_undegraded":"2021-09-16T19:37:55.477320+0000","last_fullsized":"2021-09-16T19:37:55.477320+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8692,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104848908,"statfs":{"total":107374182400,"available":107365281792,"internally_reserved":0,"allocated":700416,"data_stored":618949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7592,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850008,"statfs":{"total":107374182400,"available":107366408192,"internally_reserved":0,"allocated":98304,"data_stored":24777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8684,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104848916,"statfs":{"total":107374182400,"available":107365289984,"internally_reserved":0,"allocated":692224,"data_stored":615145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-16T19:38:00.345 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-09-16T19:38:00.626 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:38:00.628 INFO:teuthology.orchestra.run.smithi118.stderr:dumped all 2021-09-16T19:38:00.639 INFO:teuthology.orchestra.run.smithi118.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2021-09-16T19:38:00.259549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24968,"kb_used_data":1456,"kb_used_omap":0,"kb_used_meta":23232,"kb_avail":314547832,"statfs":{"total":322122547200,"available":322096979968,"internally_reserved":0,"allocated":1490944,"data_stored":1258871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23789568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000325"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.477075+0000","last_change":"2021-09-16T19:37:55.477075+0000","last_active":"2021-09-16T19:37:55.477075+0000","last_peered":"2021-09-16T19:37:55.477075+0000","last_clean":"2021-09-16T19:37:55.477075+0000","last_became_active":"2021-09-16T19:37:55.476645+0000","last_became_peered":"2021-09-16T19:37:55.476645+0000","last_unstale":"2021-09-16T19:37:55.477075+0000","last_undegraded":"2021-09-16T19:37:55.477075+0000","last_fullsized":"2021-09-16T19:37:55.477075+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.473509+0000","last_change":"2021-09-16T19:37:55.473509+0000","last_active":"2021-09-16T19:37:55.473509+0000","last_peered":"2021-09-16T19:37:55.473509+0000","last_clean":"2021-09-16T19:37:55.473509+0000","last_became_active":"2021-09-16T19:37:55.473257+0000","last_became_peered":"2021-09-16T19:37:55.473257+0000","last_unstale":"2021-09-16T19:37:55.473509+0000","last_undegraded":"2021-09-16T19:37:55.473509+0000","last_fullsized":"2021-09-16T19:37:55.473509+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.477326+0000","last_change":"2021-09-16T19:37:55.475859+0000","last_active":"2021-09-16T19:37:55.477326+0000","last_peered":"2021-09-16T19:37:55.477326+0000","last_clean":"2021-09-16T19:37:55.477326+0000","last_became_active":"2021-09-16T19:37:55.475599+0000","last_became_peered":"2021-09-16T19:37:55.475599+0000","last_unstale":"2021-09-16T19:37:55.477326+0000","last_undegraded":"2021-09-16T19:37:55.477326+0000","last_fullsized":"2021-09-16T19:37:55.477326+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.475059+0000","last_change":"2021-09-16T19:37:55.475059+0000","last_active":"2021-09-16T19:37:55.475059+0000","last_peered":"2021-09-16T19:37:55.475059+0000","last_clean":"2021-09-16T19:37:55.475059+0000","last_became_active":"2021-09-16T19:37:55.474791+0000","last_became_peered":"2021-09-16T19:37:55.474791+0000","last_unstale":"2021-09-16T19:37:55.475059+0000","last_undegraded":"2021-09-16T19:37:55.475059+0000","last_fullsized":"2021-09-16T19:37:55.475059+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.474282+0000","last_change":"2021-09-16T19:37:55.474282+0000","last_active":"2021-09-16T19:37:55.474282+0000","last_peered":"2021-09-16T19:37:55.474282+0000","last_clean":"2021-09-16T19:37:55.474282+0000","last_became_active":"2021-09-16T19:37:55.474038+0000","last_became_peered":"2021-09-16T19:37:55.474038+0000","last_unstale":"2021-09-16T19:37:55.474282+0000","last_undegraded":"2021-09-16T19:37:55.474282+0000","last_fullsized":"2021-09-16T19:37:55.474282+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"11'97","reported_seq":136,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.867873+0000","last_change":"2021-09-16T19:37:55.477001+0000","last_active":"2021-09-16T19:37:55.867873+0000","last_peered":"2021-09-16T19:37:55.867873+0000","last_clean":"2021-09-16T19:37:55.867873+0000","last_became_active":"2021-09-16T19:37:55.476216+0000","last_became_peered":"2021-09-16T19:37:55.476216+0000","last_unstale":"2021-09-16T19:37:55.867873+0000","last_undegraded":"2021-09-16T19:37:55.867873+0000","last_fullsized":"2021-09-16T19:37:55.867873+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.476649+0000","last_change":"2021-09-16T19:37:55.476649+0000","last_active":"2021-09-16T19:37:55.476649+0000","last_peered":"2021-09-16T19:37:55.476649+0000","last_clean":"2021-09-16T19:37:55.476649+0000","last_became_active":"2021-09-16T19:37:55.476390+0000","last_became_peered":"2021-09-16T19:37:55.476390+0000","last_unstale":"2021-09-16T19:37:55.476649+0000","last_undegraded":"2021-09-16T19:37:55.476649+0000","last_fullsized":"2021-09-16T19:37:55.476649+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.475889+0000","last_change":"2021-09-16T19:37:55.475889+0000","last_active":"2021-09-16T19:37:55.475889+0000","last_peered":"2021-09-16T19:37:55.475889+0000","last_clean":"2021-09-16T19:37:55.475889+0000","last_became_active":"2021-09-16T19:37:55.475637+0000","last_became_peered":"2021-09-16T19:37:55.475637+0000","last_unstale":"2021-09-16T19:37:55.475889+0000","last_undegraded":"2021-09-16T19:37:55.475889+0000","last_fullsized":"2021-09-16T19:37:55.475889+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-16T19:37:55.477320+0000","last_change":"2021-09-16T19:37:55.475884+0000","last_active":"2021-09-16T19:37:55.477320+0000","last_peered":"2021-09-16T19:37:55.477320+0000","last_clean":"2021-09-16T19:37:55.477320+0000","last_became_active":"2021-09-16T19:37:55.475637+0000","last_became_peered":"2021-09-16T19:37:55.475637+0000","last_unstale":"2021-09-16T19:37:55.477320+0000","last_undegraded":"2021-09-16T19:37:55.477320+0000","last_fullsized":"2021-09-16T19:37:55.477320+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-16T19:37:54.467755+0000","last_clean_scrub_stamp":"2021-09-16T19:37:54.467755+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8692,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104848908,"statfs":{"total":107374182400,"available":107365281792,"internally_reserved":0,"allocated":700416,"data_stored":618949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7592,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850008,"statfs":{"total":107374182400,"available":107366408192,"internally_reserved":0,"allocated":98304,"data_stored":24777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8684,"kb_used_data":676,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104848916,"statfs":{"total":107374182400,"available":107365289984,"internally_reserved":0,"allocated":692224,"data_stored":615145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.252}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-16T19:38:00.640 INFO:tasks.ceph.ceph_manager.c2:clean! 2021-09-16T19:38:00.640 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2021-09-16T19:38:00.641 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2021-09-16T19:38:00.641 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2021-09-16T19:38:00.984 INFO:teuthology.orchestra.run.smithi118.stdout: 2021-09-16T19:38:00.984 INFO:teuthology.orchestra.run.smithi118.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-09-16T19:38:00.997 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2021-09-16T19:38:00.998 INFO:teuthology.run_tasks:Running task rgw... 2021-09-16T19:38:01.014 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2021-09-16T19:38:01.014 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2021-09-16T19:38:01.015 INFO:tasks.rgw:Creating data pools 2021-09-16T19:38:01.016 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2021-09-16T19:38:01.016 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-09-16T19:38:01.817 INFO:teuthology.orchestra.run.smithi109.stderr:pool 'default.rgw.buckets.data' created 2021-09-16T19:38:01.833 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-09-16T19:38:02.828 INFO:teuthology.orchestra.run.smithi109.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-16T19:38:02.849 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-09-16T19:38:03.834 INFO:teuthology.orchestra.run.smithi109.stderr:pool 'default.rgw.buckets.index' created 2021-09-16T19:38:03.845 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-09-16T19:38:04.846 INFO:teuthology.orchestra.run.smithi109.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-16T19:38:04.862 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2021-09-16T19:38:04.863 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-09-16T19:38:05.144 INFO:teuthology.orchestra.run.smithi109.stderr:pool 'default.rgw.buckets.data' already exists 2021-09-16T19:38:05.154 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-09-16T19:38:05.830 INFO:teuthology.orchestra.run.smithi109.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-16T19:38:05.841 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-09-16T19:38:06.122 INFO:teuthology.orchestra.run.smithi109.stderr:pool 'default.rgw.buckets.index' already exists 2021-09-16T19:38:06.133 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-09-16T19:38:06.840 INFO:teuthology.orchestra.run.smithi109.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-16T19:38:06.851 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2021-09-16T19:38:06.851 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-09-16T19:38:07.492 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'default.rgw.buckets.data' created 2021-09-16T19:38:07.506 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-09-16T19:38:08.521 INFO:teuthology.orchestra.run.smithi118.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-16T19:38:08.533 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-09-16T19:38:09.506 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'default.rgw.buckets.index' created 2021-09-16T19:38:09.520 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-09-16T19:38:10.509 INFO:teuthology.orchestra.run.smithi118.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-16T19:38:10.529 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2021-09-16T19:38:10.529 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-09-16T19:38:10.813 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'default.rgw.buckets.data' already exists 2021-09-16T19:38:10.824 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-09-16T19:38:11.500 INFO:teuthology.orchestra.run.smithi118.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-16T19:38:11.511 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-09-16T19:38:11.789 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'default.rgw.buckets.index' already exists 2021-09-16T19:38:11.800 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-09-16T19:38:12.503 INFO:teuthology.orchestra.run.smithi118.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-16T19:38:12.514 DEBUG:tasks.rgw:Pools created 2021-09-16T19:38:12.515 INFO:tasks.rgw:Configuring compression type = random 2021-09-16T19:38:12.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2021-09-16T19:38:12.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2021-09-16T19:38:12.517 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2021-09-16T19:38:12.590 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:12.610 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:12.611 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:12.595+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.855 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.853+0000 7f18a114f380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:13.855 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.853+0000 7f18a114f380 20 realm 2021-09-16T19:38:13.856 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.853+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.856 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.854+0000 7f18a114f380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:13.856 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.854+0000 7f18a114f380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-16T19:38:13.857 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.854+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.857 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.855+0000 7f18a114f380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:13.857 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.855+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.860 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.858+0000 7f18a114f380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:13.860 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.858+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.860 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.858+0000 7f18a114f380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:13.860 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.859+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.861 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.859+0000 7f18a114f380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:13.861 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.859+0000 7f18a114f380 10 Creating default zonegroup 2021-09-16T19:38:13.861 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.859+0000 7f18a114f380 20 get_system_obj_state: rctx=0x7ffe651ea700 obj=:.pools.avail state=0x557e6a9d35c0 s->prefetch_data=0 2021-09-16T19:38:13.862 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.860+0000 7f18a114f380 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-09-16T19:38:13.871 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.870+0000 7f18a114f380 10 WARNING: store->list_zones() returned r=-2 2021-09-16T19:38:13.872 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.870+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.872 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.870+0000 7f18a114f380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:13.874 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.872+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.874 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.873+0000 7f18a114f380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:13.874 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.873+0000 7f18a114f380 10 could not read realm id: (2) No such file or directory 2021-09-16T19:38:13.875 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.873+0000 7f18a114f380 10 WARNING: failed to set zone as default, r=-22 2021-09-16T19:38:13.875 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.873+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.875 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.873+0000 7f18a114f380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:13.879 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.878+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.882 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.881+0000 7f18a114f380 20 rados_obj.operate() r=0 bl.length=873 2021-09-16T19:38:13.882 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.881+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.883 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.881+0000 7f18a114f380 20 rados_obj.operate() r=0 bl.length=358 2021-09-16T19:38:13.884 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.881+0000 7f18a114f380 10 Using default name default 2021-09-16T19:38:13.884 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.881+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.884 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.882+0000 7f18a114f380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:13.884 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.882+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.885 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.882+0000 7f18a114f380 20 rados_obj.operate() r=0 bl.length=873 2021-09-16T19:38:13.885 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.882+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.885 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.882+0000 7f18a114f380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:13.885 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.882+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.885 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.882+0000 7f18a114f380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:13.886 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.882+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.886 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.883+0000 7f18a114f380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:13.893 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.890+0000 7f18a114f380 20 RGWRados::pool_iterate: got zone_info.fddd76d0-b52a-4508-9264-7f1ecd0b7a55 2021-09-16T19:38:13.893 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.890+0000 7f18a114f380 20 RGWRados::pool_iterate: got zonegroup_info.c5d2a46d-eb6b-4bac-b464-569241f1bb78 2021-09-16T19:38:13.893 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.890+0000 7f18a114f380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-16T19:38:13.894 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.890+0000 7f18a114f380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-16T19:38:13.894 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.890+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.894 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.890+0000 7f18a114f380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:13.894 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.890+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.894 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.891+0000 7f18a114f380 20 rados_obj.operate() r=0 bl.length=873 2021-09-16T19:38:13.895 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.891+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.895 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.891+0000 7f18a114f380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:13.895 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.891+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.895 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.891+0000 7f18a114f380 20 rados_obj.operate() r=0 bl.length=358 2021-09-16T19:38:13.896 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.891+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.896 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.891+0000 7f18a114f380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:13.896 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.891+0000 7f18a114f380 20 zone default found 2021-09-16T19:38:13.896 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.891+0000 7f18a114f380 1 Realm: () 2021-09-16T19:38:13.896 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.891+0000 7f18a114f380 1 ZoneGroup: default (c5d2a46d-eb6b-4bac-b464-569241f1bb78) 2021-09-16T19:38:13.897 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.891+0000 7f18a114f380 1 Zone: default (fddd76d0-b52a-4508-9264-7f1ecd0b7a55) 2021-09-16T19:38:13.897 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.891+0000 7f18a114f380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-16T19:38:13.897 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.891+0000 7f18a114f380 20 zonegroup default 2021-09-16T19:38:13.897 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.891+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:13.897 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.892+0000 7f18a114f380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:13.898 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:13.892+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:15.866 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:15.865+0000 7f18a114f380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:15.867 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:15.865+0000 7f18a114f380 20 rados->read ofs=0 len=0 2021-09-16T19:38:15.868 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:15.866+0000 7f18a114f380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:15.868 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:15.866+0000 7f18a114f380 20 started sync module instance, tier type = 2021-09-16T19:38:15.868 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:15.866+0000 7f18a114f380 20 started zone id=fddd76d0-b52a-4508-9264-7f1ecd0b7a55 (name=default) with tier type = 2021-09-16T19:38:17.899 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.897+0000 7f18a114f380 20 add_watcher() i=0 2021-09-16T19:38:17.899 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.897+0000 7f18a114f380 20 add_watcher() i=1 2021-09-16T19:38:17.900 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.897+0000 7f18a114f380 20 add_watcher() i=2 2021-09-16T19:38:17.900 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.897+0000 7f18a114f380 20 add_watcher() i=3 2021-09-16T19:38:17.900 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.897+0000 7f18a114f380 20 add_watcher() i=4 2021-09-16T19:38:17.900 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.897+0000 7f18a114f380 20 add_watcher() i=5 2021-09-16T19:38:17.900 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.897+0000 7f18a114f380 20 add_watcher() i=6 2021-09-16T19:38:17.901 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.900+0000 7f18a114f380 20 add_watcher() i=7 2021-09-16T19:38:17.901 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.900+0000 7f18a114f380 2 all 8 watchers are set, enabling cache 2021-09-16T19:38:17.903 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.900+0000 7f18a114f380 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2021-09-16T19:38:17.903 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.900+0000 7f18a114f380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2021-09-16T19:38:17.903 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.900+0000 7f18a114f380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-16T19:38:17.903 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.901+0000 7f18a114f380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2021-09-16T19:38:17.904 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.901+0000 7f18a114f380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2021-09-16T19:38:17.904 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.901+0000 7f18a114f380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-16T19:38:17.904 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.901+0000 7f18a114f380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2021-09-16T19:38:17.904 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.901+0000 7f18a114f380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2021-09-16T19:38:17.905 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.901+0000 7f18a114f380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-16T19:38:17.905 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.901+0000 7f18a114f380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2021-09-16T19:38:17.905 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.901+0000 7f18a114f380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2021-09-16T19:38:17.905 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.901+0000 7f18a114f380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-16T19:38:17.906 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.901+0000 7f18a114f380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2021-09-16T19:38:17.906 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.901+0000 7f18a114f380 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2021-09-16T19:38:17.906 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.903+0000 7f18a114f380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-16T19:38:17.912 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.910+0000 7f1865fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-16T19:38:17.912 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.911+0000 7f18a114f380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-16T19:38:17.912 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:17.911+0000 7f18a114f380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-16T19:38:21.880 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.878+0000 7f18a114f380 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-09-16T19:38:21.880 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.878+0000 7f18a114f380 5 note: GC not initialized 2021-09-16T19:38:21.882 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.880+0000 7f17d4ff9700 20 reqs_thread_entry: start 2021-09-16T19:38:21.882 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.880+0000 7f17c67fc700 20 reqs_thread_entry: start 2021-09-16T19:38:21.882 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.880+0000 7f18a114f380 20 init_complete bucket index max shards: 11 2021-09-16T19:38:21.882 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.880+0000 7f18a114f380 10 rgw notify: Started notification manager with: 1 workers 2021-09-16T19:38:21.883 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.880+0000 7f17c4ff9700 20 rgw notify: INFO: next queues processing will happen at: Thu Sep 16 19:38:52 2021 2021-09-16T19:38:21.883 INFO:teuthology.orchestra.run.smithi109.stderr: 2021-09-16T19:38:21.883 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.880+0000 7f17abfff700 20 reqs_thread_entry: start 2021-09-16T19:38:21.902 INFO:teuthology.orchestra.run.smithi109.stdout:[] 2021-09-16T19:38:21.903 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.901+0000 7f18a114f380 20 remove_watcher() i=0 2021-09-16T19:38:21.903 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.901+0000 7f18a114f380 2 removed watcher, disabling cache 2021-09-16T19:38:21.903 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.902+0000 7f18a114f380 20 remove_watcher() i=1 2021-09-16T19:38:21.904 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.903+0000 7f18a114f380 20 remove_watcher() i=2 2021-09-16T19:38:21.905 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.904+0000 7f18a114f380 20 remove_watcher() i=3 2021-09-16T19:38:21.906 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.905+0000 7f18a114f380 20 remove_watcher() i=4 2021-09-16T19:38:21.911 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.910+0000 7f18a114f380 20 remove_watcher() i=5 2021-09-16T19:38:21.912 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.911+0000 7f18a114f380 20 remove_watcher() i=6 2021-09-16T19:38:21.913 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.912+0000 7f18a114f380 20 remove_watcher() i=7 2021-09-16T19:38:21.914 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.913+0000 7f1865fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-16T19:38:21.919 DEBUG:tasks.util.rgw: json result: [] 2021-09-16T19:38:21.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-16T19:38:21.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-16T19:38:21.921 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-16T19:38:21.967 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:21.967 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:21.972 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.971+0000 7fcbdb965380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-16T19:38:21.973 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.971+0000 7fcbdb965380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-16T19:38:21.976 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.975+0000 7fcb17fff700 20 reqs_thread_entry: start 2021-09-16T19:38:21.976 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.975+0000 7fcbdb965380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-16T19:38:21.976 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.975+0000 7fcbdb965380 20 rados->read ofs=0 len=0 2021-09-16T19:38:21.977 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.975+0000 7fcbdb965380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:21.977 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.975+0000 7fcbdb965380 20 rados->read ofs=0 len=0 2021-09-16T19:38:21.978 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.976+0000 7fcbdb965380 20 rados_obj.operate() r=0 bl.length=873 2021-09-16T19:38:21.985 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.983+0000 7fcbdb965380 20 RGWRados::pool_iterate: got zone_info.fddd76d0-b52a-4508-9264-7f1ecd0b7a55 2021-09-16T19:38:21.986 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.983+0000 7fcbdb965380 20 RGWRados::pool_iterate: got zonegroup_info.c5d2a46d-eb6b-4bac-b464-569241f1bb78 2021-09-16T19:38:21.986 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.983+0000 7fcbdb965380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-16T19:38:21.986 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.983+0000 7fcbdb965380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-16T19:38:21.986 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.983+0000 7fcbdb965380 20 rados->read ofs=0 len=0 2021-09-16T19:38:21.987 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.984+0000 7fcbdb965380 20 rados_obj.operate() r=0 bl.length=873 2021-09-16T19:38:21.987 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.984+0000 7fcbdb965380 20 rados->read ofs=0 len=0 2021-09-16T19:38:21.987 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.984+0000 7fcbdb965380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:21.987 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.984+0000 7fcbdb965380 20 rados->read ofs=0 len=0 2021-09-16T19:38:21.988 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.984+0000 7fcbdb965380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:21.988 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.984+0000 7fcbdb965380 20 rados->read ofs=0 len=0 2021-09-16T19:38:21.988 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:21.984+0000 7fcbdb965380 20 rados_obj.operate() r=0 bl.length=358 2021-09-16T19:38:21.989 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:38:21.990 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "fddd76d0-b52a-4508-9264-7f1ecd0b7a55", 2021-09-16T19:38:21.990 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "default", 2021-09-16T19:38:21.990 INFO:teuthology.orchestra.run.smithi109.stdout: "domain_root": "default.rgw.meta:root", 2021-09-16T19:38:21.990 INFO:teuthology.orchestra.run.smithi109.stdout: "control_pool": "default.rgw.control", 2021-09-16T19:38:21.991 INFO:teuthology.orchestra.run.smithi109.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-16T19:38:21.991 INFO:teuthology.orchestra.run.smithi109.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-16T19:38:21.991 INFO:teuthology.orchestra.run.smithi109.stdout: "log_pool": "default.rgw.log", 2021-09-16T19:38:21.991 INFO:teuthology.orchestra.run.smithi109.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-16T19:38:21.992 INFO:teuthology.orchestra.run.smithi109.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-16T19:38:21.992 INFO:teuthology.orchestra.run.smithi109.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-16T19:38:21.992 INFO:teuthology.orchestra.run.smithi109.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-16T19:38:21.992 INFO:teuthology.orchestra.run.smithi109.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-16T19:38:21.992 INFO:teuthology.orchestra.run.smithi109.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-16T19:38:21.993 INFO:teuthology.orchestra.run.smithi109.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-16T19:38:21.993 INFO:teuthology.orchestra.run.smithi109.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-16T19:38:21.993 INFO:teuthology.orchestra.run.smithi109.stdout: "otp_pool": "default.rgw.otp", 2021-09-16T19:38:21.993 INFO:teuthology.orchestra.run.smithi109.stdout: "system_key": { 2021-09-16T19:38:21.993 INFO:teuthology.orchestra.run.smithi109.stdout: "access_key": "", 2021-09-16T19:38:21.994 INFO:teuthology.orchestra.run.smithi109.stdout: "secret_key": "" 2021-09-16T19:38:21.994 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:38:21.994 INFO:teuthology.orchestra.run.smithi109.stdout: "placement_pools": [ 2021-09-16T19:38:21.994 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:38:21.995 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "default-placement", 2021-09-16T19:38:21.995 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:38:21.995 INFO:teuthology.orchestra.run.smithi109.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-16T19:38:21.995 INFO:teuthology.orchestra.run.smithi109.stdout: "storage_classes": { 2021-09-16T19:38:21.995 INFO:teuthology.orchestra.run.smithi109.stdout: "STANDARD": { 2021-09-16T19:38:21.996 INFO:teuthology.orchestra.run.smithi109.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-16T19:38:21.996 INFO:teuthology.orchestra.run.smithi109.stdout: "compression_type": "random" 2021-09-16T19:38:21.996 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:21.996 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:38:21.996 INFO:teuthology.orchestra.run.smithi109.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-16T19:38:21.997 INFO:teuthology.orchestra.run.smithi109.stdout: "index_type": 0 2021-09-16T19:38:21.997 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:21.997 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:21.997 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:38:21.997 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "", 2021-09-16T19:38:21.998 INFO:teuthology.orchestra.run.smithi109.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-16T19:38:21.998 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:38:21.999 DEBUG:tasks.util.rgw: json result: {'id': 'fddd76d0-b52a-4508-9264-7f1ecd0b7a55', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-16T19:38:21.999 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2021-09-16T19:38:22.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2021-09-16T19:38:22.000 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2021-09-16T19:38:22.038 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:22.038 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:22.043 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.042+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.044 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.043+0000 7f613e4bf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:22.045 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.043+0000 7f613e4bf380 20 realm 2021-09-16T19:38:22.045 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.043+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.045 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.043+0000 7f613e4bf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:22.045 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.043+0000 7f613e4bf380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-16T19:38:22.046 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.043+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.046 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.043+0000 7f613e4bf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:22.046 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.043+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.046 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.043+0000 7f613e4bf380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:22.047 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.043+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.047 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.044+0000 7f613e4bf380 20 rados_obj.operate() r=0 bl.length=889 2021-09-16T19:38:22.048 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.044+0000 7f613e4bf380 20 searching for the correct realm 2021-09-16T19:38:22.054 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.051+0000 7f613e4bf380 20 RGWRados::pool_iterate: got zone_info.fddd76d0-b52a-4508-9264-7f1ecd0b7a55 2021-09-16T19:38:22.054 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.051+0000 7f613e4bf380 20 RGWRados::pool_iterate: got zonegroup_info.c5d2a46d-eb6b-4bac-b464-569241f1bb78 2021-09-16T19:38:22.054 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.051+0000 7f613e4bf380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-16T19:38:22.055 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.051+0000 7f613e4bf380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-16T19:38:22.055 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.051+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.055 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.051+0000 7f613e4bf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:22.055 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.051+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.056 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.051+0000 7f613e4bf380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:22.056 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.051+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.056 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.052+0000 7f613e4bf380 20 rados_obj.operate() r=0 bl.length=358 2021-09-16T19:38:22.056 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.052+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.057 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.052+0000 7f613e4bf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:22.057 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.052+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.057 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.052+0000 7f613e4bf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:22.057 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.052+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.057 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.052+0000 7f613e4bf380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:22.063 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.058+0000 7f613e4bf380 20 RGWRados::pool_iterate: got zone_info.fddd76d0-b52a-4508-9264-7f1ecd0b7a55 2021-09-16T19:38:22.063 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.058+0000 7f613e4bf380 20 RGWRados::pool_iterate: got zonegroup_info.c5d2a46d-eb6b-4bac-b464-569241f1bb78 2021-09-16T19:38:22.063 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.058+0000 7f613e4bf380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-16T19:38:22.064 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.058+0000 7f613e4bf380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-16T19:38:22.064 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.058+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.064 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.059+0000 7f613e4bf380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:22.064 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.059+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.065 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.059+0000 7f613e4bf380 20 rados_obj.operate() r=0 bl.length=889 2021-09-16T19:38:22.065 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.059+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.065 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.059+0000 7f613e4bf380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:22.065 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.059+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.066 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.059+0000 7f613e4bf380 20 rados_obj.operate() r=0 bl.length=358 2021-09-16T19:38:22.066 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.059+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.066 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.059+0000 7f613e4bf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:22.066 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.059+0000 7f613e4bf380 20 zone default found 2021-09-16T19:38:22.066 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.059+0000 7f613e4bf380 1 Realm: () 2021-09-16T19:38:22.067 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.059+0000 7f613e4bf380 1 ZoneGroup: default (c5d2a46d-eb6b-4bac-b464-569241f1bb78) 2021-09-16T19:38:22.067 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.059+0000 7f613e4bf380 1 Zone: default (fddd76d0-b52a-4508-9264-7f1ecd0b7a55) 2021-09-16T19:38:22.067 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.059+0000 7f613e4bf380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-16T19:38:22.067 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.059+0000 7f613e4bf380 20 zonegroup default 2021-09-16T19:38:22.067 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.059+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.068 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.060+0000 7f613e4bf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:22.068 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.060+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.068 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.060+0000 7f613e4bf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:22.068 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.060+0000 7f613e4bf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.069 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.060+0000 7f613e4bf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:22.069 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.060+0000 7f613e4bf380 20 started sync module instance, tier type = 2021-09-16T19:38:22.069 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.060+0000 7f613e4bf380 20 started zone id=fddd76d0-b52a-4508-9264-7f1ecd0b7a55 (name=default) with tier type = 2021-09-16T19:38:22.083 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.082+0000 7f613e4bf380 20 add_watcher() i=0 2021-09-16T19:38:22.083 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.082+0000 7f613e4bf380 20 add_watcher() i=1 2021-09-16T19:38:22.084 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.082+0000 7f613e4bf380 20 add_watcher() i=2 2021-09-16T19:38:22.084 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.082+0000 7f613e4bf380 20 add_watcher() i=3 2021-09-16T19:38:22.084 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.082+0000 7f613e4bf380 20 add_watcher() i=4 2021-09-16T19:38:22.084 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.082+0000 7f613e4bf380 20 add_watcher() i=5 2021-09-16T19:38:22.085 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.082+0000 7f613e4bf380 20 add_watcher() i=6 2021-09-16T19:38:22.085 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.084+0000 7f613e4bf380 20 add_watcher() i=7 2021-09-16T19:38:22.086 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.084+0000 7f613e4bf380 2 all 8 watchers are set, enabling cache 2021-09-16T19:38:22.090 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.089+0000 7f61027e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-16T19:38:22.091 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.089+0000 7f613e4bf380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-16T19:38:22.091 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.089+0000 7f613e4bf380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-16T19:38:22.094 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.093+0000 7f613e4bf380 5 note: GC not initialized 2021-09-16T19:38:22.095 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.093+0000 7f60717fa700 20 reqs_thread_entry: start 2021-09-16T19:38:22.095 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.094+0000 7f606affd700 20 reqs_thread_entry: start 2021-09-16T19:38:22.096 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.094+0000 7f613e4bf380 20 init_complete bucket index max shards: 11 2021-09-16T19:38:22.096 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.094+0000 7f613e4bf380 10 rgw notify: Started notification manager with: 1 workers 2021-09-16T19:38:22.096 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.094+0000 7f60697fa700 20 rgw notify: INFO: next queues processing will happen at: Thu Sep 16 19:38:52 2021 2021-09-16T19:38:22.096 INFO:teuthology.orchestra.run.smithi109.stderr: 2021-09-16T19:38:22.097 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.094+0000 7f6068ff9700 20 reqs_thread_entry: start 2021-09-16T19:38:22.103 INFO:teuthology.orchestra.run.smithi109.stdout:[] 2021-09-16T19:38:22.103 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.101+0000 7f613e4bf380 20 remove_watcher() i=0 2021-09-16T19:38:22.103 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.101+0000 7f613e4bf380 2 removed watcher, disabling cache 2021-09-16T19:38:22.105 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.104+0000 7f613e4bf380 20 remove_watcher() i=1 2021-09-16T19:38:22.108 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.107+0000 7f613e4bf380 20 remove_watcher() i=2 2021-09-16T19:38:22.110 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.109+0000 7f613e4bf380 20 remove_watcher() i=3 2021-09-16T19:38:22.112 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.111+0000 7f613e4bf380 20 remove_watcher() i=4 2021-09-16T19:38:22.115 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.114+0000 7f613e4bf380 20 remove_watcher() i=5 2021-09-16T19:38:22.118 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.117+0000 7f613e4bf380 20 remove_watcher() i=6 2021-09-16T19:38:22.118 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.117+0000 7f613e4bf380 20 remove_watcher() i=7 2021-09-16T19:38:22.119 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.118+0000 7f61027e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-16T19:38:22.124 DEBUG:tasks.util.rgw: json result: [] 2021-09-16T19:38:22.124 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-16T19:38:22.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-16T19:38:22.125 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-16T19:38:22.207 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:22.207 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:22.214 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.210+0000 7f883fcbf380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-16T19:38:22.214 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.210+0000 7f883fcbf380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-16T19:38:22.215 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.214+0000 7f8777fff700 20 reqs_thread_entry: start 2021-09-16T19:38:22.215 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.214+0000 7f883fcbf380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-16T19:38:22.216 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.214+0000 7f883fcbf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.216 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.215+0000 7f883fcbf380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:22.217 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.215+0000 7f883fcbf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.217 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.215+0000 7f883fcbf380 20 rados_obj.operate() r=0 bl.length=889 2021-09-16T19:38:22.224 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.222+0000 7f883fcbf380 20 RGWRados::pool_iterate: got zone_info.fddd76d0-b52a-4508-9264-7f1ecd0b7a55 2021-09-16T19:38:22.224 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.222+0000 7f883fcbf380 20 RGWRados::pool_iterate: got zonegroup_info.c5d2a46d-eb6b-4bac-b464-569241f1bb78 2021-09-16T19:38:22.225 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.222+0000 7f883fcbf380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-16T19:38:22.225 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.222+0000 7f883fcbf380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-16T19:38:22.225 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.222+0000 7f883fcbf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.225 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.222+0000 7f883fcbf380 20 rados_obj.operate() r=0 bl.length=889 2021-09-16T19:38:22.225 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.222+0000 7f883fcbf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.226 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.223+0000 7f883fcbf380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:22.226 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.223+0000 7f883fcbf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.226 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.223+0000 7f883fcbf380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:22.226 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.223+0000 7f883fcbf380 20 rados->read ofs=0 len=0 2021-09-16T19:38:22.227 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:22.223+0000 7f883fcbf380 20 rados_obj.operate() r=0 bl.length=358 2021-09-16T19:38:22.228 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:38:22.228 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "fddd76d0-b52a-4508-9264-7f1ecd0b7a55", 2021-09-16T19:38:22.228 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "default", 2021-09-16T19:38:22.228 INFO:teuthology.orchestra.run.smithi109.stdout: "domain_root": "default.rgw.meta:root", 2021-09-16T19:38:22.229 INFO:teuthology.orchestra.run.smithi109.stdout: "control_pool": "default.rgw.control", 2021-09-16T19:38:22.229 INFO:teuthology.orchestra.run.smithi109.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-16T19:38:22.229 INFO:teuthology.orchestra.run.smithi109.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-16T19:38:22.229 INFO:teuthology.orchestra.run.smithi109.stdout: "log_pool": "default.rgw.log", 2021-09-16T19:38:22.229 INFO:teuthology.orchestra.run.smithi109.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-16T19:38:22.230 INFO:teuthology.orchestra.run.smithi109.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-16T19:38:22.230 INFO:teuthology.orchestra.run.smithi109.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-16T19:38:22.230 INFO:teuthology.orchestra.run.smithi109.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-16T19:38:22.230 INFO:teuthology.orchestra.run.smithi109.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-16T19:38:22.231 INFO:teuthology.orchestra.run.smithi109.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-16T19:38:22.231 INFO:teuthology.orchestra.run.smithi109.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-16T19:38:22.231 INFO:teuthology.orchestra.run.smithi109.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-16T19:38:22.231 INFO:teuthology.orchestra.run.smithi109.stdout: "otp_pool": "default.rgw.otp", 2021-09-16T19:38:22.231 INFO:teuthology.orchestra.run.smithi109.stdout: "system_key": { 2021-09-16T19:38:22.232 INFO:teuthology.orchestra.run.smithi109.stdout: "access_key": "", 2021-09-16T19:38:22.232 INFO:teuthology.orchestra.run.smithi109.stdout: "secret_key": "" 2021-09-16T19:38:22.232 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:38:22.232 INFO:teuthology.orchestra.run.smithi109.stdout: "placement_pools": [ 2021-09-16T19:38:22.233 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:38:22.233 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "default-placement", 2021-09-16T19:38:22.233 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:38:22.233 INFO:teuthology.orchestra.run.smithi109.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-16T19:38:22.233 INFO:teuthology.orchestra.run.smithi109.stdout: "storage_classes": { 2021-09-16T19:38:22.234 INFO:teuthology.orchestra.run.smithi109.stdout: "STANDARD": { 2021-09-16T19:38:22.234 INFO:teuthology.orchestra.run.smithi109.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-16T19:38:22.234 INFO:teuthology.orchestra.run.smithi109.stdout: "compression_type": "random" 2021-09-16T19:38:22.234 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:22.234 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:38:22.235 INFO:teuthology.orchestra.run.smithi109.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-16T19:38:22.235 INFO:teuthology.orchestra.run.smithi109.stdout: "index_type": 0 2021-09-16T19:38:22.235 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:22.235 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:22.236 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:38:22.236 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "", 2021-09-16T19:38:22.236 INFO:teuthology.orchestra.run.smithi109.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-16T19:38:22.236 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:38:22.237 DEBUG:tasks.util.rgw: json result: {'id': 'fddd76d0-b52a-4508-9264-7f1ecd0b7a55', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-16T19:38:22.238 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2021-09-16T19:38:22.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2021-09-16T19:38:22.238 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2021-09-16T19:38:22.310 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:22.311 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:22.316 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:22.315+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.527 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.526+0000 7ff14b82b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:23.528 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.526+0000 7ff14b82b380 20 realm 2021-09-16T19:38:23.528 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.526+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.530 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.528+0000 7ff14b82b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:23.530 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.528+0000 7ff14b82b380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-16T19:38:23.531 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.528+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.531 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.529+0000 7ff14b82b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:23.531 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.529+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.531 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.529+0000 7ff14b82b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:23.532 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.529+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.533 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.529+0000 7ff14b82b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:23.533 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.529+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.533 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.529+0000 7ff14b82b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:23.533 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.529+0000 7ff14b82b380 10 Creating default zonegroup 2021-09-16T19:38:23.534 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.530+0000 7ff14b82b380 20 get_system_obj_state: rctx=0x7ffd047ad7b0 obj=:.pools.avail state=0x5574b01de5c0 s->prefetch_data=0 2021-09-16T19:38:23.534 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.530+0000 7ff14b82b380 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-09-16T19:38:23.541 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.539+0000 7ff14b82b380 10 WARNING: store->list_zones() returned r=-2 2021-09-16T19:38:23.541 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.539+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.541 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.539+0000 7ff14b82b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:23.543 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.541+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.543 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.542+0000 7ff14b82b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:23.544 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.542+0000 7ff14b82b380 10 could not read realm id: (2) No such file or directory 2021-09-16T19:38:23.544 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.542+0000 7ff14b82b380 10 WARNING: failed to set zone as default, r=-22 2021-09-16T19:38:23.544 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.542+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.544 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.542+0000 7ff14b82b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:23.545 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.544+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.546 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.544+0000 7ff14b82b380 20 rados_obj.operate() r=0 bl.length=873 2021-09-16T19:38:23.546 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.544+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.547 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.544+0000 7ff14b82b380 20 rados_obj.operate() r=0 bl.length=358 2021-09-16T19:38:23.547 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.544+0000 7ff14b82b380 10 Using default name default 2021-09-16T19:38:23.547 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.544+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.547 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.545+0000 7ff14b82b380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:23.548 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.545+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.548 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.545+0000 7ff14b82b380 20 rados_obj.operate() r=0 bl.length=873 2021-09-16T19:38:23.548 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.545+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.549 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.545+0000 7ff14b82b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:23.549 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.545+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.549 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.545+0000 7ff14b82b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:23.549 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.545+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.550 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.546+0000 7ff14b82b380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:23.556 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.553+0000 7ff14b82b380 20 RGWRados::pool_iterate: got zone_info.bc0f6638-a64b-4faa-9ee8-66bb27d7927f 2021-09-16T19:38:23.556 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.553+0000 7ff14b82b380 20 RGWRados::pool_iterate: got zonegroup_info.ab282862-91f8-466d-9e08-da7a2b5007cb 2021-09-16T19:38:23.556 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.553+0000 7ff14b82b380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-16T19:38:23.557 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.553+0000 7ff14b82b380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-16T19:38:23.557 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.553+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.557 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.553+0000 7ff14b82b380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:23.557 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.553+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.558 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.554+0000 7ff14b82b380 20 rados_obj.operate() r=0 bl.length=873 2021-09-16T19:38:23.558 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.554+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.558 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.554+0000 7ff14b82b380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:23.558 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.554+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.559 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.554+0000 7ff14b82b380 20 rados_obj.operate() r=0 bl.length=358 2021-09-16T19:38:23.559 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.554+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.559 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.554+0000 7ff14b82b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:23.559 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.554+0000 7ff14b82b380 20 zone default found 2021-09-16T19:38:23.560 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.554+0000 7ff14b82b380 1 Realm: () 2021-09-16T19:38:23.560 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.554+0000 7ff14b82b380 1 ZoneGroup: default (ab282862-91f8-466d-9e08-da7a2b5007cb) 2021-09-16T19:38:23.560 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.554+0000 7ff14b82b380 1 Zone: default (bc0f6638-a64b-4faa-9ee8-66bb27d7927f) 2021-09-16T19:38:23.560 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.554+0000 7ff14b82b380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-16T19:38:23.560 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.554+0000 7ff14b82b380 20 zonegroup default 2021-09-16T19:38:23.561 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.554+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:23.561 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.555+0000 7ff14b82b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:23.561 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:23.555+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:25.537 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:25.535+0000 7ff14b82b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:25.537 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:25.535+0000 7ff14b82b380 20 rados->read ofs=0 len=0 2021-09-16T19:38:25.537 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:25.535+0000 7ff14b82b380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:25.537 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:25.535+0000 7ff14b82b380 20 started sync module instance, tier type = 2021-09-16T19:38:25.538 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:25.535+0000 7ff14b82b380 20 started zone id=bc0f6638-a64b-4faa-9ee8-66bb27d7927f (name=default) with tier type = 2021-09-16T19:38:27.554 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.552+0000 7ff14b82b380 20 add_watcher() i=0 2021-09-16T19:38:27.554 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.552+0000 7ff14b82b380 20 add_watcher() i=1 2021-09-16T19:38:27.555 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.552+0000 7ff14b82b380 20 add_watcher() i=2 2021-09-16T19:38:27.555 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.552+0000 7ff14b82b380 20 add_watcher() i=3 2021-09-16T19:38:27.555 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.552+0000 7ff14b82b380 20 add_watcher() i=4 2021-09-16T19:38:27.555 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.552+0000 7ff14b82b380 20 add_watcher() i=5 2021-09-16T19:38:27.555 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.552+0000 7ff14b82b380 20 add_watcher() i=6 2021-09-16T19:38:27.557 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.553+0000 7ff14b82b380 20 add_watcher() i=7 2021-09-16T19:38:27.557 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.553+0000 7ff14b82b380 2 all 8 watchers are set, enabling cache 2021-09-16T19:38:27.557 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.553+0000 7ff14b82b380 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2021-09-16T19:38:27.558 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.553+0000 7ff14b82b380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2021-09-16T19:38:27.558 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.553+0000 7ff14b82b380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-16T19:38:27.558 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.553+0000 7ff14b82b380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2021-09-16T19:38:27.558 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.553+0000 7ff14b82b380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2021-09-16T19:38:27.559 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.553+0000 7ff14b82b380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-16T19:38:27.559 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.554+0000 7ff14b82b380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2021-09-16T19:38:27.559 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.554+0000 7ff14b82b380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2021-09-16T19:38:27.559 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.554+0000 7ff14b82b380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-16T19:38:27.559 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.554+0000 7ff14b82b380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2021-09-16T19:38:27.560 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.554+0000 7ff14b82b380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2021-09-16T19:38:27.560 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.554+0000 7ff14b82b380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-16T19:38:27.560 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.554+0000 7ff14b82b380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2021-09-16T19:38:27.560 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.554+0000 7ff14b82b380 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2021-09-16T19:38:27.561 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.555+0000 7ff14b82b380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-16T19:38:27.561 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.557+0000 7ff087fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-16T19:38:27.562 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.557+0000 7ff14b82b380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-16T19:38:27.562 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:27.557+0000 7ff14b82b380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-16T19:38:31.539 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.538+0000 7ff14b82b380 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-09-16T19:38:31.540 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.538+0000 7ff14b82b380 5 note: GC not initialized 2021-09-16T19:38:31.541 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.539+0000 7ff076ffd700 20 reqs_thread_entry: start 2021-09-16T19:38:31.541 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.539+0000 7ff074ff9700 20 reqs_thread_entry: start 2021-09-16T19:38:31.541 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.539+0000 7ff14b82b380 20 init_complete bucket index max shards: 11 2021-09-16T19:38:31.541 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.539+0000 7ff066ffd700 20 rgw notify: INFO: next queues processing will happen at: Thu Sep 16 19:39:01 2021 2021-09-16T19:38:31.542 INFO:teuthology.orchestra.run.smithi118.stderr: 2021-09-16T19:38:31.542 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.540+0000 7ff14b82b380 10 rgw notify: Started notification manager with: 1 workers 2021-09-16T19:38:31.543 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.540+0000 7ff0667fc700 20 reqs_thread_entry: start 2021-09-16T19:38:31.561 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2021-09-16T19:38:31.562 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.560+0000 7ff14b82b380 20 remove_watcher() i=0 2021-09-16T19:38:31.562 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.560+0000 7ff14b82b380 2 removed watcher, disabling cache 2021-09-16T19:38:31.563 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.561+0000 7ff14b82b380 20 remove_watcher() i=1 2021-09-16T19:38:31.563 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.562+0000 7ff14b82b380 20 remove_watcher() i=2 2021-09-16T19:38:31.564 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.563+0000 7ff14b82b380 20 remove_watcher() i=3 2021-09-16T19:38:31.565 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.564+0000 7ff14b82b380 20 remove_watcher() i=4 2021-09-16T19:38:31.566 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.565+0000 7ff14b82b380 20 remove_watcher() i=5 2021-09-16T19:38:31.567 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.566+0000 7ff14b82b380 20 remove_watcher() i=6 2021-09-16T19:38:31.568 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.567+0000 7ff14b82b380 20 remove_watcher() i=7 2021-09-16T19:38:31.569 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.568+0000 7ff087fff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-16T19:38:31.575 DEBUG:tasks.util.rgw: json result: [] 2021-09-16T19:38:31.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-16T19:38:31.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-16T19:38:31.576 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-16T19:38:31.623 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:31.623 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:31.629 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.627+0000 7f9ddee74380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-16T19:38:31.629 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.627+0000 7f9ddee74380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-16T19:38:31.632 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.631+0000 7f9d1affd700 20 reqs_thread_entry: start 2021-09-16T19:38:31.633 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.631+0000 7f9ddee74380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-16T19:38:31.633 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.631+0000 7f9ddee74380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.634 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.632+0000 7f9ddee74380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:31.634 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.632+0000 7f9ddee74380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.634 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.632+0000 7f9ddee74380 20 rados_obj.operate() r=0 bl.length=873 2021-09-16T19:38:31.642 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.640+0000 7f9ddee74380 20 RGWRados::pool_iterate: got zone_info.bc0f6638-a64b-4faa-9ee8-66bb27d7927f 2021-09-16T19:38:31.643 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.640+0000 7f9ddee74380 20 RGWRados::pool_iterate: got zonegroup_info.ab282862-91f8-466d-9e08-da7a2b5007cb 2021-09-16T19:38:31.643 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.640+0000 7f9ddee74380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-16T19:38:31.643 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.640+0000 7f9ddee74380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-16T19:38:31.643 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.640+0000 7f9ddee74380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.644 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.640+0000 7f9ddee74380 20 rados_obj.operate() r=0 bl.length=873 2021-09-16T19:38:31.644 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.640+0000 7f9ddee74380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.644 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.641+0000 7f9ddee74380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:31.644 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.641+0000 7f9ddee74380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.644 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.641+0000 7f9ddee74380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:31.645 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.641+0000 7f9ddee74380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.645 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.641+0000 7f9ddee74380 20 rados_obj.operate() r=0 bl.length=358 2021-09-16T19:38:31.646 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:38:31.647 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "bc0f6638-a64b-4faa-9ee8-66bb27d7927f", 2021-09-16T19:38:31.647 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default", 2021-09-16T19:38:31.647 INFO:teuthology.orchestra.run.smithi118.stdout: "domain_root": "default.rgw.meta:root", 2021-09-16T19:38:31.647 INFO:teuthology.orchestra.run.smithi118.stdout: "control_pool": "default.rgw.control", 2021-09-16T19:38:31.647 INFO:teuthology.orchestra.run.smithi118.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-16T19:38:31.648 INFO:teuthology.orchestra.run.smithi118.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-16T19:38:31.648 INFO:teuthology.orchestra.run.smithi118.stdout: "log_pool": "default.rgw.log", 2021-09-16T19:38:31.648 INFO:teuthology.orchestra.run.smithi118.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-16T19:38:31.648 INFO:teuthology.orchestra.run.smithi118.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-16T19:38:31.649 INFO:teuthology.orchestra.run.smithi118.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-16T19:38:31.649 INFO:teuthology.orchestra.run.smithi118.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-16T19:38:31.649 INFO:teuthology.orchestra.run.smithi118.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-16T19:38:31.649 INFO:teuthology.orchestra.run.smithi118.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-16T19:38:31.649 INFO:teuthology.orchestra.run.smithi118.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-16T19:38:31.650 INFO:teuthology.orchestra.run.smithi118.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-16T19:38:31.650 INFO:teuthology.orchestra.run.smithi118.stdout: "otp_pool": "default.rgw.otp", 2021-09-16T19:38:31.650 INFO:teuthology.orchestra.run.smithi118.stdout: "system_key": { 2021-09-16T19:38:31.650 INFO:teuthology.orchestra.run.smithi118.stdout: "access_key": "", 2021-09-16T19:38:31.651 INFO:teuthology.orchestra.run.smithi118.stdout: "secret_key": "" 2021-09-16T19:38:31.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:38:31.651 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_pools": [ 2021-09-16T19:38:31.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:38:31.651 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "default-placement", 2021-09-16T19:38:31.652 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:38:31.652 INFO:teuthology.orchestra.run.smithi118.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-16T19:38:31.652 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": { 2021-09-16T19:38:31.652 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD": { 2021-09-16T19:38:31.652 INFO:teuthology.orchestra.run.smithi118.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-16T19:38:31.653 INFO:teuthology.orchestra.run.smithi118.stdout: "compression_type": "random" 2021-09-16T19:38:31.653 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:38:31.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:38:31.653 INFO:teuthology.orchestra.run.smithi118.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-16T19:38:31.653 INFO:teuthology.orchestra.run.smithi118.stdout: "index_type": 0 2021-09-16T19:38:31.654 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:38:31.654 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:38:31.654 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:38:31.654 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "", 2021-09-16T19:38:31.655 INFO:teuthology.orchestra.run.smithi118.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-16T19:38:31.655 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:38:31.656 DEBUG:tasks.util.rgw: json result: {'id': 'bc0f6638-a64b-4faa-9ee8-66bb27d7927f', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-16T19:38:31.656 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2021-09-16T19:38:31.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2021-09-16T19:38:31.657 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2021-09-16T19:38:31.695 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:31.696 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:31.701 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.699+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.702 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.700+0000 7fd6f2a30380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:31.703 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.700+0000 7fd6f2a30380 20 realm 2021-09-16T19:38:31.703 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.700+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.703 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.701+0000 7fd6f2a30380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:31.704 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.701+0000 7fd6f2a30380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-16T19:38:31.704 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.701+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.704 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.701+0000 7fd6f2a30380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:31.705 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.701+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.705 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.701+0000 7fd6f2a30380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:31.705 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.701+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.706 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.702+0000 7fd6f2a30380 20 rados_obj.operate() r=0 bl.length=889 2021-09-16T19:38:31.706 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.702+0000 7fd6f2a30380 20 searching for the correct realm 2021-09-16T19:38:31.711 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.709+0000 7fd6f2a30380 20 RGWRados::pool_iterate: got zone_info.bc0f6638-a64b-4faa-9ee8-66bb27d7927f 2021-09-16T19:38:31.711 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.709+0000 7fd6f2a30380 20 RGWRados::pool_iterate: got zonegroup_info.ab282862-91f8-466d-9e08-da7a2b5007cb 2021-09-16T19:38:31.712 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.709+0000 7fd6f2a30380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-16T19:38:31.712 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.709+0000 7fd6f2a30380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-16T19:38:31.712 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.709+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.712 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.709+0000 7fd6f2a30380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:31.713 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.709+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.713 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.709+0000 7fd6f2a30380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:31.713 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.709+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.713 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.709+0000 7fd6f2a30380 20 rados_obj.operate() r=0 bl.length=358 2021-09-16T19:38:31.713 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.709+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.714 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.709+0000 7fd6f2a30380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:31.714 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.710+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.714 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.710+0000 7fd6f2a30380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:31.714 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.710+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.715 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.710+0000 7fd6f2a30380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:31.719 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.716+0000 7fd6f2a30380 20 RGWRados::pool_iterate: got zone_info.bc0f6638-a64b-4faa-9ee8-66bb27d7927f 2021-09-16T19:38:31.719 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.716+0000 7fd6f2a30380 20 RGWRados::pool_iterate: got zonegroup_info.ab282862-91f8-466d-9e08-da7a2b5007cb 2021-09-16T19:38:31.720 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.716+0000 7fd6f2a30380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-16T19:38:31.720 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.716+0000 7fd6f2a30380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-16T19:38:31.720 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.716+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.720 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.716+0000 7fd6f2a30380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:31.721 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.716+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.721 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.716+0000 7fd6f2a30380 20 rados_obj.operate() r=0 bl.length=889 2021-09-16T19:38:31.721 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.716+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.721 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.716+0000 7fd6f2a30380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:31.721 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.716+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.722 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.717+0000 7fd6f2a30380 20 rados_obj.operate() r=0 bl.length=358 2021-09-16T19:38:31.722 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.717+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.722 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.717+0000 7fd6f2a30380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:31.722 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.717+0000 7fd6f2a30380 20 zone default found 2021-09-16T19:38:31.723 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.717+0000 7fd6f2a30380 1 Realm: () 2021-09-16T19:38:31.723 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.717+0000 7fd6f2a30380 1 ZoneGroup: default (ab282862-91f8-466d-9e08-da7a2b5007cb) 2021-09-16T19:38:31.723 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.717+0000 7fd6f2a30380 1 Zone: default (bc0f6638-a64b-4faa-9ee8-66bb27d7927f) 2021-09-16T19:38:31.723 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.717+0000 7fd6f2a30380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-16T19:38:31.723 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.717+0000 7fd6f2a30380 20 zonegroup default 2021-09-16T19:38:31.724 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.717+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.724 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.717+0000 7fd6f2a30380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:31.724 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.717+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.724 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.717+0000 7fd6f2a30380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:31.724 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.717+0000 7fd6f2a30380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.725 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.718+0000 7fd6f2a30380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:31.725 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.718+0000 7fd6f2a30380 20 started sync module instance, tier type = 2021-09-16T19:38:31.725 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.718+0000 7fd6f2a30380 20 started zone id=bc0f6638-a64b-4faa-9ee8-66bb27d7927f (name=default) with tier type = 2021-09-16T19:38:31.727 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.725+0000 7fd6f2a30380 20 add_watcher() i=0 2021-09-16T19:38:31.727 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.725+0000 7fd6f2a30380 20 add_watcher() i=1 2021-09-16T19:38:31.727 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.725+0000 7fd6f2a30380 20 add_watcher() i=2 2021-09-16T19:38:31.727 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.725+0000 7fd6f2a30380 20 add_watcher() i=3 2021-09-16T19:38:31.728 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.725+0000 7fd6f2a30380 20 add_watcher() i=4 2021-09-16T19:38:31.728 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.725+0000 7fd6f2a30380 20 add_watcher() i=5 2021-09-16T19:38:31.728 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.725+0000 7fd6f2a30380 20 add_watcher() i=6 2021-09-16T19:38:31.729 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.726+0000 7fd6f2a30380 20 add_watcher() i=7 2021-09-16T19:38:31.729 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.726+0000 7fd6f2a30380 2 all 8 watchers are set, enabling cache 2021-09-16T19:38:31.729 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.727+0000 7fd6b6fe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-16T19:38:31.729 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.727+0000 7fd6f2a30380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-16T19:38:31.730 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.727+0000 7fd6f2a30380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-16T19:38:31.732 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.731+0000 7fd6f2a30380 5 note: GC not initialized 2021-09-16T19:38:31.733 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.731+0000 7fd625ffb700 20 reqs_thread_entry: start 2021-09-16T19:38:31.734 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.732+0000 7fd6177fe700 20 reqs_thread_entry: start 2021-09-16T19:38:31.734 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.732+0000 7fd6f2a30380 20 init_complete bucket index max shards: 11 2021-09-16T19:38:31.734 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.732+0000 7fd6f2a30380 10 rgw notify: Started notification manager with: 1 workers 2021-09-16T19:38:31.734 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.732+0000 7fd615ffb700 20 rgw notify: INFO: next queues processing will happen at: Thu Sep 16 19:39:02 2021 2021-09-16T19:38:31.735 INFO:teuthology.orchestra.run.smithi118.stderr: 2021-09-16T19:38:31.735 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.732+0000 7fd6157fa700 20 reqs_thread_entry: start 2021-09-16T19:38:31.741 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2021-09-16T19:38:31.741 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.739+0000 7fd6f2a30380 20 remove_watcher() i=0 2021-09-16T19:38:31.741 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.739+0000 7fd6f2a30380 2 removed watcher, disabling cache 2021-09-16T19:38:31.742 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.740+0000 7fd6f2a30380 20 remove_watcher() i=1 2021-09-16T19:38:31.743 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.741+0000 7fd6f2a30380 20 remove_watcher() i=2 2021-09-16T19:38:31.743 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.742+0000 7fd6f2a30380 20 remove_watcher() i=3 2021-09-16T19:38:31.744 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.743+0000 7fd6f2a30380 20 remove_watcher() i=4 2021-09-16T19:38:31.745 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.744+0000 7fd6f2a30380 20 remove_watcher() i=5 2021-09-16T19:38:31.746 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.744+0000 7fd6f2a30380 20 remove_watcher() i=6 2021-09-16T19:38:31.746 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.745+0000 7fd6f2a30380 20 remove_watcher() i=7 2021-09-16T19:38:31.747 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.746+0000 7fd6b6fe5700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-16T19:38:31.751 DEBUG:tasks.util.rgw: json result: [] 2021-09-16T19:38:31.752 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-16T19:38:31.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-16T19:38:31.753 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-16T19:38:31.835 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:31.836 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:31.840 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.839+0000 7fecec520380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-16T19:38:31.841 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.839+0000 7fecec520380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-16T19:38:31.844 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.842+0000 7fec23fff700 20 reqs_thread_entry: start 2021-09-16T19:38:31.844 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.842+0000 7fecec520380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-16T19:38:31.844 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.842+0000 7fecec520380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.845 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.843+0000 7fecec520380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:31.846 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.843+0000 7fecec520380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.846 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.844+0000 7fecec520380 20 rados_obj.operate() r=0 bl.length=889 2021-09-16T19:38:31.852 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.850+0000 7fecec520380 20 RGWRados::pool_iterate: got zone_info.bc0f6638-a64b-4faa-9ee8-66bb27d7927f 2021-09-16T19:38:31.853 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.850+0000 7fecec520380 20 RGWRados::pool_iterate: got zonegroup_info.ab282862-91f8-466d-9e08-da7a2b5007cb 2021-09-16T19:38:31.853 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.850+0000 7fecec520380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-16T19:38:31.854 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.850+0000 7fecec520380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-16T19:38:31.854 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.850+0000 7fecec520380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.854 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.851+0000 7fecec520380 20 rados_obj.operate() r=0 bl.length=889 2021-09-16T19:38:31.854 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.851+0000 7fecec520380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.855 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.851+0000 7fecec520380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-16T19:38:31.855 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.851+0000 7fecec520380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.856 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.851+0000 7fecec520380 20 rados_obj.operate() r=0 bl.length=46 2021-09-16T19:38:31.856 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.851+0000 7fecec520380 20 rados->read ofs=0 len=0 2021-09-16T19:38:31.856 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:31.851+0000 7fecec520380 20 rados_obj.operate() r=0 bl.length=358 2021-09-16T19:38:31.857 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:38:31.857 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "bc0f6638-a64b-4faa-9ee8-66bb27d7927f", 2021-09-16T19:38:31.857 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default", 2021-09-16T19:38:31.857 INFO:teuthology.orchestra.run.smithi118.stdout: "domain_root": "default.rgw.meta:root", 2021-09-16T19:38:31.857 INFO:teuthology.orchestra.run.smithi118.stdout: "control_pool": "default.rgw.control", 2021-09-16T19:38:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-16T19:38:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-16T19:38:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: "log_pool": "default.rgw.log", 2021-09-16T19:38:31.858 INFO:teuthology.orchestra.run.smithi118.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-16T19:38:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-16T19:38:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-16T19:38:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-16T19:38:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-16T19:38:31.859 INFO:teuthology.orchestra.run.smithi118.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-16T19:38:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-16T19:38:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-16T19:38:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: "otp_pool": "default.rgw.otp", 2021-09-16T19:38:31.860 INFO:teuthology.orchestra.run.smithi118.stdout: "system_key": { 2021-09-16T19:38:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: "access_key": "", 2021-09-16T19:38:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: "secret_key": "" 2021-09-16T19:38:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:38:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_pools": [ 2021-09-16T19:38:31.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:38:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "default-placement", 2021-09-16T19:38:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:38:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-16T19:38:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": { 2021-09-16T19:38:31.862 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD": { 2021-09-16T19:38:31.863 INFO:teuthology.orchestra.run.smithi118.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-16T19:38:31.863 INFO:teuthology.orchestra.run.smithi118.stdout: "compression_type": "random" 2021-09-16T19:38:31.864 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:38:31.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:38:31.864 INFO:teuthology.orchestra.run.smithi118.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-16T19:38:31.864 INFO:teuthology.orchestra.run.smithi118.stdout: "index_type": 0 2021-09-16T19:38:31.865 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:38:31.865 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:38:31.865 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:38:31.865 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "", 2021-09-16T19:38:31.865 INFO:teuthology.orchestra.run.smithi118.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-16T19:38:31.866 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:38:31.867 DEBUG:tasks.util.rgw: json result: {'id': 'bc0f6638-a64b-4faa-9ee8-66bb27d7927f', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-16T19:38:31.867 INFO:tasks.rgw:Starting rgw... 2021-09-16T19:38:31.868 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2021-09-16T19:38:31.868 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-16T19:38:31.869 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-09-16T19:38:31.869 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-09-16T19:38:31.872 INFO:tasks.rgw.c1.client.0:Started 2021-09-16T19:38:31.873 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2021-09-16T19:38:31.873 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-16T19:38:31.873 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-09-16T19:38:31.874 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-09-16T19:38:31.876 INFO:tasks.rgw.c1.client.1:Started 2021-09-16T19:38:31.876 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2021-09-16T19:38:31.877 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-16T19:38:31.877 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-09-16T19:38:31.877 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-09-16T19:38:31.910 INFO:tasks.rgw.c2.client.0:Started 2021-09-16T19:38:31.910 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2021-09-16T19:38:31.910 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-16T19:38:31.911 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-09-16T19:38:31.911 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-09-16T19:38:31.913 INFO:tasks.rgw.c2.client.1:Started 2021-09-16T19:38:31.914 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi109.front.sepia.ceph.com:8000/ 2021-09-16T19:38:31.914 DEBUG:teuthology.orchestra.run.smithi109:> curl http://smithi109.front.sepia.ceph.com:8000/ 2021-09-16T19:38:31.949 INFO:teuthology.orchestra.run.smithi109.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-16T19:38:31.949 INFO:teuthology.orchestra.run.smithi109.stderr: Dload Upload Total Spent Left Speed 2021-09-16T19:38:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-16T19:38:31.951 INFO:teuthology.orchestra.run.smithi109.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi109.front.sepia.ceph.com port 8000: Connection refused 2021-09-16T19:38:32.952 DEBUG:teuthology.orchestra.run.smithi109:> curl http://smithi109.front.sepia.ceph.com:8000/ 2021-09-16T19:38:32.971 INFO:teuthology.orchestra.run.smithi109.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-16T19:38:32.972 INFO:teuthology.orchestra.run.smithi109.stderr: Dload Upload Total Spent Left Speed 2021-09-16T19:38:32.973 INFO:teuthology.orchestra.run.smithi109.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2021-09-16T19:38:32.974 INFO:teuthology.orchestra.run.smithi109.stdout:anonymous 2021-09-16T19:38:32.974 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi109.front.sepia.ceph.com:8001/ 2021-09-16T19:38:32.974 DEBUG:teuthology.orchestra.run.smithi109:> curl http://smithi109.front.sepia.ceph.com:8001/ 2021-09-16T19:38:33.032 INFO:teuthology.orchestra.run.smithi109.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-16T19:38:33.033 INFO:teuthology.orchestra.run.smithi109.stderr: Dload Upload Total Spent Left Speed 2021-09-16T19:38:33.034 INFO:teuthology.orchestra.run.smithi109.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 208k 2021-09-16T19:38:33.035 INFO:teuthology.orchestra.run.smithi109.stdout:anonymous 2021-09-16T19:38:33.035 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi118.front.sepia.ceph.com:8000/ 2021-09-16T19:38:33.036 DEBUG:teuthology.orchestra.run.smithi118:> curl http://smithi118.front.sepia.ceph.com:8000/ 2021-09-16T19:38:33.071 INFO:teuthology.orchestra.run.smithi118.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-16T19:38:33.071 INFO:teuthology.orchestra.run.smithi118.stderr: Dload Upload Total Spent Left Speed 2021-09-16T19:38:33.073 INFO:teuthology.orchestra.run.smithi118.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2021-09-16T19:38:33.074 INFO:teuthology.orchestra.run.smithi118.stdout:anonymous 2021-09-16T19:38:33.075 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi118.front.sepia.ceph.com:8001/ 2021-09-16T19:38:33.075 DEBUG:teuthology.orchestra.run.smithi118:> curl http://smithi118.front.sepia.ceph.com:8001/ 2021-09-16T19:38:33.145 INFO:teuthology.orchestra.run.smithi118.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-16T19:38:33.145 INFO:teuthology.orchestra.run.smithi118.stderr: Dload Upload Total Spent Left Speed 2021-09-16T19:38:33.146 INFO:teuthology.orchestra.run.smithi118.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2021-09-16T19:38:33.147 INFO:teuthology.orchestra.run.smithi118.stdout:anonymous 2021-09-16T19:38:33.147 INFO:teuthology.run_tasks:Running task rgw-multisite... 2021-09-16T19:38:33.294 INFO:tasks.rgw_multisite:creating realm.. 2021-09-16T19:38:33.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:33.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:33.296 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-16T19:38:33.344 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:33.345 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:33.371 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:38:33.371 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T19:38:33.371 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-realm", 2021-09-16T19:38:33.372 INFO:teuthology.orchestra.run.smithi109.stdout: "current_period": "6be2ae27-983c-4bba-b75f-41d101865b26", 2021-09-16T19:38:33.372 INFO:teuthology.orchestra.run.smithi109.stdout: "epoch": 1 2021-09-16T19:38:33.372 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:38:33.374 INFO:tasks.rgw_multisite:creating master zonegroup.. 2021-09-16T19:38:33.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:33.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:33.375 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-16T19:38:33.461 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:33.461 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:33.472 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:33.470+0000 7f4d6515c380 -1 old format 2021-09-16T19:38:33.479 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:38:33.479 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "test-zonegroup", 2021-09-16T19:38:33.480 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zonegroup", 2021-09-16T19:38:33.480 INFO:teuthology.orchestra.run.smithi109.stdout: "api_name": "", 2021-09-16T19:38:33.480 INFO:teuthology.orchestra.run.smithi109.stdout: "is_master": "true", 2021-09-16T19:38:33.480 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [ 2021-09-16T19:38:33.480 INFO:teuthology.orchestra.run.smithi109.stdout: "http://smithi109.front.sepia.ceph.com:8000" 2021-09-16T19:38:33.481 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:38:33.481 INFO:teuthology.orchestra.run.smithi109.stdout: "hostnames": [], 2021-09-16T19:38:33.481 INFO:teuthology.orchestra.run.smithi109.stdout: "hostnames_s3website": [], 2021-09-16T19:38:33.481 INFO:teuthology.orchestra.run.smithi109.stdout: "master_zone": "", 2021-09-16T19:38:33.482 INFO:teuthology.orchestra.run.smithi109.stdout: "zones": [], 2021-09-16T19:38:33.482 INFO:teuthology.orchestra.run.smithi109.stdout: "placement_targets": [], 2021-09-16T19:38:33.482 INFO:teuthology.orchestra.run.smithi109.stdout: "default_placement": "", 2021-09-16T19:38:33.482 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T19:38:33.483 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_policy": { 2021-09-16T19:38:33.483 INFO:teuthology.orchestra.run.smithi109.stdout: "groups": [] 2021-09-16T19:38:33.483 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:33.483 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:38:33.484 INFO:tasks.rgw_multisite:creating master zone.. 2021-09-16T19:38:33.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'K0MH58GXSJSVNR3I', '--secret', '93vysIyO7YxIyUlpTN7GyE2EQAEwRODi', '--endpoints', 'http://smithi109.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:33.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'K0MH58GXSJSVNR3I', '--secret', '93vysIyO7YxIyUlpTN7GyE2EQAEwRODi', '--endpoints', 'http://smithi109.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:33.485 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key K0MH58GXSJSVNR3I --secret 93vysIyO7YxIyUlpTN7GyE2EQAEwRODi --endpoints http://smithi109.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-16T19:38:33.570 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:33.570 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:33.598 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:38:33.599 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:38:33.599 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zone1", 2021-09-16T19:38:33.599 INFO:teuthology.orchestra.run.smithi109.stdout: "domain_root": "test-zone1.rgw.meta:root", 2021-09-16T19:38:33.600 INFO:teuthology.orchestra.run.smithi109.stdout: "control_pool": "test-zone1.rgw.control", 2021-09-16T19:38:33.600 INFO:teuthology.orchestra.run.smithi109.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2021-09-16T19:38:33.600 INFO:teuthology.orchestra.run.smithi109.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2021-09-16T19:38:33.600 INFO:teuthology.orchestra.run.smithi109.stdout: "log_pool": "test-zone1.rgw.log", 2021-09-16T19:38:33.600 INFO:teuthology.orchestra.run.smithi109.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2021-09-16T19:38:33.601 INFO:teuthology.orchestra.run.smithi109.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2021-09-16T19:38:33.601 INFO:teuthology.orchestra.run.smithi109.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2021-09-16T19:38:33.601 INFO:teuthology.orchestra.run.smithi109.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2021-09-16T19:38:33.601 INFO:teuthology.orchestra.run.smithi109.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2021-09-16T19:38:33.602 INFO:teuthology.orchestra.run.smithi109.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2021-09-16T19:38:33.602 INFO:teuthology.orchestra.run.smithi109.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2021-09-16T19:38:33.602 INFO:teuthology.orchestra.run.smithi109.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2021-09-16T19:38:33.602 INFO:teuthology.orchestra.run.smithi109.stdout: "otp_pool": "test-zone1.rgw.otp", 2021-09-16T19:38:33.603 INFO:teuthology.orchestra.run.smithi109.stdout: "system_key": { 2021-09-16T19:38:33.603 INFO:teuthology.orchestra.run.smithi109.stdout: "access_key": "K0MH58GXSJSVNR3I", 2021-09-16T19:38:33.603 INFO:teuthology.orchestra.run.smithi109.stdout: "secret_key": "93vysIyO7YxIyUlpTN7GyE2EQAEwRODi" 2021-09-16T19:38:33.603 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:38:33.603 INFO:teuthology.orchestra.run.smithi109.stdout: "placement_pools": [ 2021-09-16T19:38:33.604 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:38:33.604 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "default-placement", 2021-09-16T19:38:33.604 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:38:33.604 INFO:teuthology.orchestra.run.smithi109.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2021-09-16T19:38:33.604 INFO:teuthology.orchestra.run.smithi109.stdout: "storage_classes": { 2021-09-16T19:38:33.605 INFO:teuthology.orchestra.run.smithi109.stdout: "STANDARD": { 2021-09-16T19:38:33.605 INFO:teuthology.orchestra.run.smithi109.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2021-09-16T19:38:33.605 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:33.605 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:38:33.606 INFO:teuthology.orchestra.run.smithi109.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2021-09-16T19:38:33.606 INFO:teuthology.orchestra.run.smithi109.stdout: "index_type": 0 2021-09-16T19:38:33.606 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:33.606 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:33.607 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:38:33.607 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T19:38:33.607 INFO:teuthology.orchestra.run.smithi109.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2021-09-16T19:38:33.607 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:38:33.609 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2021-09-16T19:38:34.904 INFO:teuthology.orchestra.run.smithi109.stderr:pool 'test-zone1.rgw.buckets.data' created 2021-09-16T19:38:34.923 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2021-09-16T19:38:35.924 INFO:teuthology.orchestra.run.smithi109.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2021-09-16T19:38:35.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:35.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:35.940 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-16T19:38:35.985 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:35.985 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:36.008 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:38:36.009 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:38:36.009 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zone1", 2021-09-16T19:38:36.009 INFO:teuthology.orchestra.run.smithi109.stdout: "domain_root": "test-zone1.rgw.meta:root", 2021-09-16T19:38:36.009 INFO:teuthology.orchestra.run.smithi109.stdout: "control_pool": "test-zone1.rgw.control", 2021-09-16T19:38:36.010 INFO:teuthology.orchestra.run.smithi109.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2021-09-16T19:38:36.010 INFO:teuthology.orchestra.run.smithi109.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2021-09-16T19:38:36.010 INFO:teuthology.orchestra.run.smithi109.stdout: "log_pool": "test-zone1.rgw.log", 2021-09-16T19:38:36.010 INFO:teuthology.orchestra.run.smithi109.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2021-09-16T19:38:36.010 INFO:teuthology.orchestra.run.smithi109.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2021-09-16T19:38:36.011 INFO:teuthology.orchestra.run.smithi109.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2021-09-16T19:38:36.011 INFO:teuthology.orchestra.run.smithi109.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2021-09-16T19:38:36.011 INFO:teuthology.orchestra.run.smithi109.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2021-09-16T19:38:36.011 INFO:teuthology.orchestra.run.smithi109.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2021-09-16T19:38:36.012 INFO:teuthology.orchestra.run.smithi109.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2021-09-16T19:38:36.012 INFO:teuthology.orchestra.run.smithi109.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2021-09-16T19:38:36.012 INFO:teuthology.orchestra.run.smithi109.stdout: "otp_pool": "test-zone1.rgw.otp", 2021-09-16T19:38:36.012 INFO:teuthology.orchestra.run.smithi109.stdout: "system_key": { 2021-09-16T19:38:36.012 INFO:teuthology.orchestra.run.smithi109.stdout: "access_key": "K0MH58GXSJSVNR3I", 2021-09-16T19:38:36.013 INFO:teuthology.orchestra.run.smithi109.stdout: "secret_key": "93vysIyO7YxIyUlpTN7GyE2EQAEwRODi" 2021-09-16T19:38:36.013 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:38:36.013 INFO:teuthology.orchestra.run.smithi109.stdout: "placement_pools": [ 2021-09-16T19:38:36.013 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:38:36.013 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "default-placement", 2021-09-16T19:38:36.014 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:38:36.014 INFO:teuthology.orchestra.run.smithi109.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2021-09-16T19:38:36.014 INFO:teuthology.orchestra.run.smithi109.stdout: "storage_classes": { 2021-09-16T19:38:36.014 INFO:teuthology.orchestra.run.smithi109.stdout: "STANDARD": { 2021-09-16T19:38:36.015 INFO:teuthology.orchestra.run.smithi109.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2021-09-16T19:38:36.015 INFO:teuthology.orchestra.run.smithi109.stdout: "compression_type": "random" 2021-09-16T19:38:36.015 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:36.015 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:38:36.015 INFO:teuthology.orchestra.run.smithi109.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2021-09-16T19:38:36.016 INFO:teuthology.orchestra.run.smithi109.stdout: "index_type": 0 2021-09-16T19:38:36.016 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:36.016 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:36.016 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:38:36.016 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "", 2021-09-16T19:38:36.017 INFO:teuthology.orchestra.run.smithi109.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2021-09-16T19:38:36.017 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:38:36.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:36.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:36.019 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-16T19:38:36.060 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:36.060 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:36.068 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:38:36.067+0000 7f49205d6380 0 period (6be2ae27-983c-4bba-b75f-41d101865b26 does not have zone 6395206f-e97e-464f-b7bd-c10485d78bf5 configured 2021-09-16T19:38:43.985 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:38:43.986 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:38:43.986 INFO:teuthology.orchestra.run.smithi109.stdout: "epoch": 1, 2021-09-16T19:38:43.986 INFO:teuthology.orchestra.run.smithi109.stdout: "predecessor_uuid": "6be2ae27-983c-4bba-b75f-41d101865b26", 2021-09-16T19:38:43.986 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": [], 2021-09-16T19:38:43.987 INFO:teuthology.orchestra.run.smithi109.stdout: "period_map": { 2021-09-16T19:38:43.987 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:38:43.987 INFO:teuthology.orchestra.run.smithi109.stdout: "zonegroups": [ 2021-09-16T19:38:43.987 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:38:43.988 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "test-zonegroup", 2021-09-16T19:38:43.988 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zonegroup", 2021-09-16T19:38:43.988 INFO:teuthology.orchestra.run.smithi109.stdout: "api_name": "", 2021-09-16T19:38:43.988 INFO:teuthology.orchestra.run.smithi109.stdout: "is_master": "true", 2021-09-16T19:38:43.988 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [ 2021-09-16T19:38:43.989 INFO:teuthology.orchestra.run.smithi109.stdout: "http://smithi109.front.sepia.ceph.com:8000" 2021-09-16T19:38:43.989 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:38:43.989 INFO:teuthology.orchestra.run.smithi109.stdout: "hostnames": [], 2021-09-16T19:38:43.989 INFO:teuthology.orchestra.run.smithi109.stdout: "hostnames_s3website": [], 2021-09-16T19:38:43.990 INFO:teuthology.orchestra.run.smithi109.stdout: "master_zone": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:38:43.990 INFO:teuthology.orchestra.run.smithi109.stdout: "zones": [ 2021-09-16T19:38:43.990 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:38:43.990 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:38:43.990 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zone1", 2021-09-16T19:38:43.991 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [ 2021-09-16T19:38:43.991 INFO:teuthology.orchestra.run.smithi109.stdout: "http://smithi109.front.sepia.ceph.com:8000" 2021-09-16T19:38:43.991 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:38:43.991 INFO:teuthology.orchestra.run.smithi109.stdout: "log_meta": "false", 2021-09-16T19:38:43.992 INFO:teuthology.orchestra.run.smithi109.stdout: "log_data": "false", 2021-09-16T19:38:43.992 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_index_max_shards": 11, 2021-09-16T19:38:43.992 INFO:teuthology.orchestra.run.smithi109.stdout: "read_only": "false", 2021-09-16T19:38:43.992 INFO:teuthology.orchestra.run.smithi109.stdout: "tier_type": "", 2021-09-16T19:38:43.992 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from_all": "true", 2021-09-16T19:38:43.993 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from": [], 2021-09-16T19:38:43.993 INFO:teuthology.orchestra.run.smithi109.stdout: "redirect_zone": "" 2021-09-16T19:38:43.993 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:43.993 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:38:43.994 INFO:teuthology.orchestra.run.smithi109.stdout: "placement_targets": [ 2021-09-16T19:38:43.994 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:38:43.994 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "default-placement", 2021-09-16T19:38:43.994 INFO:teuthology.orchestra.run.smithi109.stdout: "tags": [], 2021-09-16T19:38:43.994 INFO:teuthology.orchestra.run.smithi109.stdout: "storage_classes": [ 2021-09-16T19:38:43.995 INFO:teuthology.orchestra.run.smithi109.stdout: "STANDARD" 2021-09-16T19:38:43.995 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:38:43.995 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:43.995 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:38:43.996 INFO:teuthology.orchestra.run.smithi109.stdout: "default_placement": "default-placement", 2021-09-16T19:38:43.996 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T19:38:43.996 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_policy": { 2021-09-16T19:38:43.996 INFO:teuthology.orchestra.run.smithi109.stdout: "groups": [] 2021-09-16T19:38:43.996 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:43.997 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:43.997 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:38:43.997 INFO:teuthology.orchestra.run.smithi109.stdout: "short_zone_ids": [ 2021-09-16T19:38:43.997 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:38:43.997 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:38:43.998 INFO:teuthology.orchestra.run.smithi109.stdout: "val": 3703685179 2021-09-16T19:38:43.998 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:43.998 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:38:43.998 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:38:43.999 INFO:teuthology.orchestra.run.smithi109.stdout: "master_zonegroup": "test-zonegroup", 2021-09-16T19:38:43.999 INFO:teuthology.orchestra.run.smithi109.stdout: "master_zone": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:38:43.999 INFO:teuthology.orchestra.run.smithi109.stdout: "period_config": { 2021-09-16T19:38:44.000 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_quota": { 2021-09-16T19:38:44.000 INFO:teuthology.orchestra.run.smithi109.stdout: "enabled": false, 2021-09-16T19:38:44.000 INFO:teuthology.orchestra.run.smithi109.stdout: "check_on_raw": false, 2021-09-16T19:38:44.000 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size": -1, 2021-09-16T19:38:44.000 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size_kb": 0, 2021-09-16T19:38:44.001 INFO:teuthology.orchestra.run.smithi109.stdout: "max_objects": -1 2021-09-16T19:38:44.001 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:38:44.001 INFO:teuthology.orchestra.run.smithi109.stdout: "user_quota": { 2021-09-16T19:38:44.001 INFO:teuthology.orchestra.run.smithi109.stdout: "enabled": false, 2021-09-16T19:38:44.002 INFO:teuthology.orchestra.run.smithi109.stdout: "check_on_raw": false, 2021-09-16T19:38:44.002 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size": -1, 2021-09-16T19:38:44.002 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size_kb": 0, 2021-09-16T19:38:44.002 INFO:teuthology.orchestra.run.smithi109.stdout: "max_objects": -1 2021-09-16T19:38:44.002 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:44.003 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:38:44.003 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T19:38:44.003 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_name": "test-realm", 2021-09-16T19:38:44.003 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:38:44.004 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:38:44.005 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-09-16T19:38:44.005 INFO:teuthology.orchestra.run:waiting for 300 2021-09-16T19:38:44.046 INFO:tasks.rgw.c1.client.0.smithi109.stdout:2021-09-16T19:38:44.045+0000 7fceb3e93700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 36594) UID: 0 2021-09-16T19:38:44.047 INFO:tasks.rgw.c1.client.0.smithi109.stdout:2021-09-16T19:38:44.045+0000 7fcec4d36440 -1 shutting down 2021-09-16T19:38:47.838 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-09-16T19:38:50.108 INFO:tasks.rgw.c1.client.0:Stopped 2021-09-16T19:38:50.108 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-09-16T19:38:50.109 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-09-16T19:38:50.111 INFO:tasks.rgw.c1.client.0:Started 2021-09-16T19:38:50.112 DEBUG:teuthology.orchestra.run.smithi109:> curl http://smithi109.front.sepia.ceph.com:8000 2021-09-16T19:38:50.133 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-16T19:38:50.134 INFO:teuthology.orchestra.run.smithi109.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-16T19:38:50.134 INFO:teuthology.orchestra.run.smithi109.stderr: Dload Upload Total Spent Left Speed 2021-09-16T19:38:50.134 INFO:teuthology.orchestra.run.smithi109.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi109.front.sepia.ceph.com port 8000: Connection refused 2021-09-16T19:38:51.135 DEBUG:teuthology.orchestra.run.smithi109:> curl http://smithi109.front.sepia.ceph.com:8000 2021-09-16T19:38:51.154 INFO:teuthology.orchestra.run.smithi109.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-16T19:38:51.154 INFO:teuthology.orchestra.run.smithi109.stderr: Dload Upload Total Spent Left Speed 2021-09-16T19:38:51.161 INFO:teuthology.orchestra.run.smithi109.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 26750 0 --:--:-- --:--:-- --:--:-- 26750 2021-09-16T19:38:51.162 INFO:teuthology.orchestra.run.smithi109.stdout:anonymous 2021-09-16T19:38:51.163 INFO:tasks.rgw_multisite:creating admin user.. 2021-09-16T19:38:51.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'K0MH58GXSJSVNR3I', '--secret', '93vysIyO7YxIyUlpTN7GyE2EQAEwRODi', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:51.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'K0MH58GXSJSVNR3I', '--secret', '93vysIyO7YxIyUlpTN7GyE2EQAEwRODi', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:51.164 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key K0MH58GXSJSVNR3I --secret 93vysIyO7YxIyUlpTN7GyE2EQAEwRODi --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-16T19:38:51.252 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:51.252 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:51.306 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:38:51.306 INFO:teuthology.orchestra.run.smithi109.stdout: "user_id": "realm-admin", 2021-09-16T19:38:51.307 INFO:teuthology.orchestra.run.smithi109.stdout: "display_name": "Realm Admin", 2021-09-16T19:38:51.307 INFO:teuthology.orchestra.run.smithi109.stdout: "email": "", 2021-09-16T19:38:51.307 INFO:teuthology.orchestra.run.smithi109.stdout: "suspended": 0, 2021-09-16T19:38:51.307 INFO:teuthology.orchestra.run.smithi109.stdout: "max_buckets": 1000, 2021-09-16T19:38:51.307 INFO:teuthology.orchestra.run.smithi109.stdout: "subusers": [], 2021-09-16T19:38:51.308 INFO:teuthology.orchestra.run.smithi109.stdout: "keys": [ 2021-09-16T19:38:51.308 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:38:51.308 INFO:teuthology.orchestra.run.smithi109.stdout: "user": "realm-admin", 2021-09-16T19:38:51.308 INFO:teuthology.orchestra.run.smithi109.stdout: "access_key": "K0MH58GXSJSVNR3I", 2021-09-16T19:38:51.308 INFO:teuthology.orchestra.run.smithi109.stdout: "secret_key": "93vysIyO7YxIyUlpTN7GyE2EQAEwRODi" 2021-09-16T19:38:51.309 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:38:51.309 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:38:51.309 INFO:teuthology.orchestra.run.smithi109.stdout: "swift_keys": [], 2021-09-16T19:38:51.309 INFO:teuthology.orchestra.run.smithi109.stdout: "caps": [], 2021-09-16T19:38:51.309 INFO:teuthology.orchestra.run.smithi109.stdout: "op_mask": "read, write, delete", 2021-09-16T19:38:51.310 INFO:teuthology.orchestra.run.smithi109.stdout: "system": "true", 2021-09-16T19:38:51.310 INFO:teuthology.orchestra.run.smithi109.stdout: "default_placement": "", 2021-09-16T19:38:51.310 INFO:teuthology.orchestra.run.smithi109.stdout: "default_storage_class": "", 2021-09-16T19:38:51.310 INFO:teuthology.orchestra.run.smithi109.stdout: "placement_tags": [], 2021-09-16T19:38:51.311 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_quota": { 2021-09-16T19:38:51.311 INFO:teuthology.orchestra.run.smithi109.stdout: "enabled": false, 2021-09-16T19:38:51.311 INFO:teuthology.orchestra.run.smithi109.stdout: "check_on_raw": false, 2021-09-16T19:38:51.311 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size": -1, 2021-09-16T19:38:51.311 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size_kb": 0, 2021-09-16T19:38:51.312 INFO:teuthology.orchestra.run.smithi109.stdout: "max_objects": -1 2021-09-16T19:38:51.312 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:38:51.312 INFO:teuthology.orchestra.run.smithi109.stdout: "user_quota": { 2021-09-16T19:38:51.312 INFO:teuthology.orchestra.run.smithi109.stdout: "enabled": false, 2021-09-16T19:38:51.312 INFO:teuthology.orchestra.run.smithi109.stdout: "check_on_raw": false, 2021-09-16T19:38:51.313 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size": -1, 2021-09-16T19:38:51.313 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size_kb": 0, 2021-09-16T19:38:51.313 INFO:teuthology.orchestra.run.smithi109.stdout: "max_objects": -1 2021-09-16T19:38:51.313 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:38:51.313 INFO:teuthology.orchestra.run.smithi109.stdout: "temp_url_keys": [], 2021-09-16T19:38:51.314 INFO:teuthology.orchestra.run.smithi109.stdout: "type": "rgw", 2021-09-16T19:38:51.314 INFO:teuthology.orchestra.run.smithi109.stdout: "mfa_ids": [] 2021-09-16T19:38:51.314 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:38:51.314 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:38:51.323 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-09-16T19:38:51.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi109.front.sepia.ceph.com:8000', '--access-key', 'K0MH58GXSJSVNR3I', '--secret', '93vysIyO7YxIyUlpTN7GyE2EQAEwRODi', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:51.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi109.front.sepia.ceph.com:8000', '--access-key', 'K0MH58GXSJSVNR3I', '--secret', '93vysIyO7YxIyUlpTN7GyE2EQAEwRODi', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:51.324 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi109.front.sepia.ceph.com:8000 --access-key K0MH58GXSJSVNR3I --secret 93vysIyO7YxIyUlpTN7GyE2EQAEwRODi --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-16T19:38:51.375 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:51.375 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:51.459 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:38:51.460 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T19:38:51.460 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-realm", 2021-09-16T19:38:51.460 INFO:teuthology.orchestra.run.smithi118.stdout: "current_period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:38:51.460 INFO:teuthology.orchestra.run.smithi118.stdout: "epoch": 2 2021-09-16T19:38:51.460 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:38:51.461 INFO:tasks.rgw_multisite:creating zone.. 2021-09-16T19:38:51.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'K0MH58GXSJSVNR3I', '--secret', '93vysIyO7YxIyUlpTN7GyE2EQAEwRODi', '--endpoints', 'http://smithi118.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:51.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'K0MH58GXSJSVNR3I', '--secret', '93vysIyO7YxIyUlpTN7GyE2EQAEwRODi', '--endpoints', 'http://smithi118.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:51.463 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key K0MH58GXSJSVNR3I --secret 93vysIyO7YxIyUlpTN7GyE2EQAEwRODi --endpoints http://smithi118.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-16T19:38:51.514 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:51.515 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:51.538 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:51.536+0000 7f6504266380 0 failed reading obj info from .rgw.root:zone_info.6395206f-e97e-464f-b7bd-c10485d78bf5: (2) No such file or directory 2021-09-16T19:38:51.538 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:51.536+0000 7f6504266380 0 WARNING: could not read zone params for zone id=6395206f-e97e-464f-b7bd-c10485d78bf5 name=test-zone1 2021-09-16T19:38:51.543 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:38:51.543 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "ceba8a70-b881-4808-8acf-e8ddfa5f6daf", 2021-09-16T19:38:51.544 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zone2", 2021-09-16T19:38:51.544 INFO:teuthology.orchestra.run.smithi118.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-16T19:38:51.545 INFO:teuthology.orchestra.run.smithi118.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-16T19:38:51.545 INFO:teuthology.orchestra.run.smithi118.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-16T19:38:51.545 INFO:teuthology.orchestra.run.smithi118.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-16T19:38:51.546 INFO:teuthology.orchestra.run.smithi118.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-16T19:38:51.546 INFO:teuthology.orchestra.run.smithi118.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-16T19:38:51.547 INFO:teuthology.orchestra.run.smithi118.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-16T19:38:51.547 INFO:teuthology.orchestra.run.smithi118.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-16T19:38:51.548 INFO:teuthology.orchestra.run.smithi118.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-16T19:38:51.548 INFO:teuthology.orchestra.run.smithi118.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-16T19:38:51.548 INFO:teuthology.orchestra.run.smithi118.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-16T19:38:51.549 INFO:teuthology.orchestra.run.smithi118.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-16T19:38:51.549 INFO:teuthology.orchestra.run.smithi118.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-16T19:38:51.550 INFO:teuthology.orchestra.run.smithi118.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-16T19:38:51.550 INFO:teuthology.orchestra.run.smithi118.stdout: "system_key": { 2021-09-16T19:38:51.551 INFO:teuthology.orchestra.run.smithi118.stdout: "access_key": "K0MH58GXSJSVNR3I", 2021-09-16T19:38:51.551 INFO:teuthology.orchestra.run.smithi118.stdout: "secret_key": "93vysIyO7YxIyUlpTN7GyE2EQAEwRODi" 2021-09-16T19:38:51.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:38:51.552 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_pools": [ 2021-09-16T19:38:51.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:38:51.553 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "default-placement", 2021-09-16T19:38:51.553 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:38:51.554 INFO:teuthology.orchestra.run.smithi118.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-16T19:38:51.554 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": { 2021-09-16T19:38:51.555 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD": { 2021-09-16T19:38:51.555 INFO:teuthology.orchestra.run.smithi118.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2021-09-16T19:38:51.555 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:38:51.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:38:51.556 INFO:teuthology.orchestra.run.smithi118.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-16T19:38:51.557 INFO:teuthology.orchestra.run.smithi118.stdout: "index_type": 0 2021-09-16T19:38:51.557 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:38:51.558 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:38:51.558 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:38:51.558 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T19:38:51.559 INFO:teuthology.orchestra.run.smithi118.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-16T19:38:51.559 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:38:51.561 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2021-09-16T19:38:52.573 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'test-zone2.rgw.buckets.data' created 2021-09-16T19:38:52.587 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2021-09-16T19:38:53.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-09-16T19:38:53.592 INFO:teuthology.orchestra.run.smithi118.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2021-09-16T19:38:53.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:53.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:53.605 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-16T19:38:53.650 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:53.651 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:53.673 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:38:53.673 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "ceba8a70-b881-4808-8acf-e8ddfa5f6daf", 2021-09-16T19:38:53.673 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zone2", 2021-09-16T19:38:53.674 INFO:teuthology.orchestra.run.smithi118.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-16T19:38:53.674 INFO:teuthology.orchestra.run.smithi118.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-16T19:38:53.674 INFO:teuthology.orchestra.run.smithi118.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-16T19:38:53.675 INFO:teuthology.orchestra.run.smithi118.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-16T19:38:53.675 INFO:teuthology.orchestra.run.smithi118.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-16T19:38:53.675 INFO:teuthology.orchestra.run.smithi118.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-16T19:38:53.676 INFO:teuthology.orchestra.run.smithi118.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-16T19:38:53.676 INFO:teuthology.orchestra.run.smithi118.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-16T19:38:53.676 INFO:teuthology.orchestra.run.smithi118.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-16T19:38:53.677 INFO:teuthology.orchestra.run.smithi118.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-16T19:38:53.677 INFO:teuthology.orchestra.run.smithi118.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-16T19:38:53.677 INFO:teuthology.orchestra.run.smithi118.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-16T19:38:53.677 INFO:teuthology.orchestra.run.smithi118.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-16T19:38:53.678 INFO:teuthology.orchestra.run.smithi118.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-16T19:38:53.678 INFO:teuthology.orchestra.run.smithi118.stdout: "system_key": { 2021-09-16T19:38:53.678 INFO:teuthology.orchestra.run.smithi118.stdout: "access_key": "K0MH58GXSJSVNR3I", 2021-09-16T19:38:53.678 INFO:teuthology.orchestra.run.smithi118.stdout: "secret_key": "93vysIyO7YxIyUlpTN7GyE2EQAEwRODi" 2021-09-16T19:38:53.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:38:53.679 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_pools": [ 2021-09-16T19:38:53.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:38:53.679 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "default-placement", 2021-09-16T19:38:53.679 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:38:53.680 INFO:teuthology.orchestra.run.smithi118.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-16T19:38:53.680 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": { 2021-09-16T19:38:53.680 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD": { 2021-09-16T19:38:53.680 INFO:teuthology.orchestra.run.smithi118.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2021-09-16T19:38:53.681 INFO:teuthology.orchestra.run.smithi118.stdout: "compression_type": "random" 2021-09-16T19:38:53.681 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:38:53.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:38:53.681 INFO:teuthology.orchestra.run.smithi118.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-16T19:38:53.682 INFO:teuthology.orchestra.run.smithi118.stdout: "index_type": 0 2021-09-16T19:38:53.682 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:38:53.682 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:38:53.682 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:38:53.683 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "", 2021-09-16T19:38:53.683 INFO:teuthology.orchestra.run.smithi118.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-16T19:38:53.683 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:38:53.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:53.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:38:53.685 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-16T19:38:53.726 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:38:53.727 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:38:53.735 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:38:53.733+0000 7f492c1ca380 0 period (db0b3d90-59a0-4d0b-99ed-8408dd9cf54f does not have zone ceba8a70-b881-4808-8acf-e8ddfa5f6daf configured 2021-09-16T19:39:01.635 INFO:teuthology.orchestra.run.smithi118.stderr:Sending period to new master zone 6395206f-e97e-464f-b7bd-c10485d78bf5 2021-09-16T19:39:01.671 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:39:01.671 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:39:01.672 INFO:teuthology.orchestra.run.smithi118.stdout: "epoch": 2, 2021-09-16T19:39:01.672 INFO:teuthology.orchestra.run.smithi118.stdout: "predecessor_uuid": "6be2ae27-983c-4bba-b75f-41d101865b26", 2021-09-16T19:39:01.672 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": [], 2021-09-16T19:39:01.672 INFO:teuthology.orchestra.run.smithi118.stdout: "period_map": { 2021-09-16T19:39:01.673 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:39:01.673 INFO:teuthology.orchestra.run.smithi118.stdout: "zonegroups": [ 2021-09-16T19:39:01.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:01.673 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "test-zonegroup", 2021-09-16T19:39:01.674 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zonegroup", 2021-09-16T19:39:01.674 INFO:teuthology.orchestra.run.smithi118.stdout: "api_name": "", 2021-09-16T19:39:01.674 INFO:teuthology.orchestra.run.smithi118.stdout: "is_master": "true", 2021-09-16T19:39:01.674 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2021-09-16T19:39:01.674 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi109.front.sepia.ceph.com:8000" 2021-09-16T19:39:01.675 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:39:01.675 INFO:teuthology.orchestra.run.smithi118.stdout: "hostnames": [], 2021-09-16T19:39:01.675 INFO:teuthology.orchestra.run.smithi118.stdout: "hostnames_s3website": [], 2021-09-16T19:39:01.675 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zone": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:39:01.676 INFO:teuthology.orchestra.run.smithi118.stdout: "zones": [ 2021-09-16T19:39:01.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:01.676 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:39:01.676 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zone1", 2021-09-16T19:39:01.676 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2021-09-16T19:39:01.677 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi109.front.sepia.ceph.com:8000" 2021-09-16T19:39:01.677 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:39:01.678 INFO:teuthology.orchestra.run.smithi118.stdout: "log_meta": "false", 2021-09-16T19:39:01.678 INFO:teuthology.orchestra.run.smithi118.stdout: "log_data": "true", 2021-09-16T19:39:01.678 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_index_max_shards": 11, 2021-09-16T19:39:01.679 INFO:teuthology.orchestra.run.smithi118.stdout: "read_only": "false", 2021-09-16T19:39:01.679 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_type": "", 2021-09-16T19:39:01.680 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from_all": "true", 2021-09-16T19:39:01.680 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from": [], 2021-09-16T19:39:01.680 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_zone": "" 2021-09-16T19:39:01.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:01.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:01.681 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "ceba8a70-b881-4808-8acf-e8ddfa5f6daf", 2021-09-16T19:39:01.681 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zone2", 2021-09-16T19:39:01.682 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2021-09-16T19:39:01.682 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2021-09-16T19:39:01.682 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:39:01.682 INFO:teuthology.orchestra.run.smithi118.stdout: "log_meta": "false", 2021-09-16T19:39:01.682 INFO:teuthology.orchestra.run.smithi118.stdout: "log_data": "true", 2021-09-16T19:39:01.683 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_index_max_shards": 11, 2021-09-16T19:39:01.683 INFO:teuthology.orchestra.run.smithi118.stdout: "read_only": "false", 2021-09-16T19:39:01.683 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_type": "", 2021-09-16T19:39:01.683 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from_all": "true", 2021-09-16T19:39:01.683 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from": [], 2021-09-16T19:39:01.684 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_zone": "" 2021-09-16T19:39:01.684 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:01.684 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:39:01.684 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_targets": [ 2021-09-16T19:39:01.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:01.685 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default-placement", 2021-09-16T19:39:01.685 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2021-09-16T19:39:01.685 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": [ 2021-09-16T19:39:01.685 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD" 2021-09-16T19:39:01.686 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:39:01.686 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:01.686 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:39:01.686 INFO:teuthology.orchestra.run.smithi118.stdout: "default_placement": "default-placement", 2021-09-16T19:39:01.686 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T19:39:01.687 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_policy": { 2021-09-16T19:39:01.687 INFO:teuthology.orchestra.run.smithi118.stdout: "groups": [] 2021-09-16T19:39:01.687 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:01.688 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:01.688 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:39:01.688 INFO:teuthology.orchestra.run.smithi118.stdout: "short_zone_ids": [ 2021-09-16T19:39:01.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:01.689 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:39:01.689 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 3703685179 2021-09-16T19:39:01.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:01.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:01.690 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "ceba8a70-b881-4808-8acf-e8ddfa5f6daf", 2021-09-16T19:39:01.690 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 3287816121 2021-09-16T19:39:01.690 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:01.691 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:39:01.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:01.691 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zonegroup": "test-zonegroup", 2021-09-16T19:39:01.691 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zone": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:39:01.691 INFO:teuthology.orchestra.run.smithi118.stdout: "period_config": { 2021-09-16T19:39:01.692 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_quota": { 2021-09-16T19:39:01.692 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": false, 2021-09-16T19:39:01.692 INFO:teuthology.orchestra.run.smithi118.stdout: "check_on_raw": false, 2021-09-16T19:39:01.692 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size": -1, 2021-09-16T19:39:01.693 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size_kb": 0, 2021-09-16T19:39:01.693 INFO:teuthology.orchestra.run.smithi118.stdout: "max_objects": -1 2021-09-16T19:39:01.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:01.693 INFO:teuthology.orchestra.run.smithi118.stdout: "user_quota": { 2021-09-16T19:39:01.694 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": false, 2021-09-16T19:39:01.694 INFO:teuthology.orchestra.run.smithi118.stdout: "check_on_raw": false, 2021-09-16T19:39:01.694 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size": -1, 2021-09-16T19:39:01.694 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size_kb": 0, 2021-09-16T19:39:01.695 INFO:teuthology.orchestra.run.smithi118.stdout: "max_objects": -1 2021-09-16T19:39:01.695 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:01.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:01.695 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T19:39:01.695 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_name": "test-realm", 2021-09-16T19:39:01.696 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:01.696 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:39:01.697 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-09-16T19:39:01.698 INFO:teuthology.orchestra.run:waiting for 300 2021-09-16T19:39:01.739 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2021-09-16T19:39:01.737+0000 7fbd760f7700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 36430) UID: 0 2021-09-16T19:39:01.739 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2021-09-16T19:39:01.737+0000 7fbd86f9a440 -1 shutting down 2021-09-16T19:39:03.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-09-16T19:39:07.801 INFO:tasks.rgw.c2.client.0:Stopped 2021-09-16T19:39:07.801 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-09-16T19:39:07.802 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-09-16T19:39:07.805 INFO:tasks.rgw.c2.client.0:Started 2021-09-16T19:39:07.806 DEBUG:teuthology.orchestra.run.smithi118:> curl http://smithi118.front.sepia.ceph.com:8000 2021-09-16T19:39:07.827 INFO:teuthology.orchestra.run.smithi118.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-16T19:39:07.828 INFO:teuthology.orchestra.run.smithi118.stderr: Dload Upload Total Spent Left Speed 2021-09-16T19:39:07.829 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-16T19:39:07.830 INFO:teuthology.orchestra.run.smithi118.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi118.front.sepia.ceph.com port 8000: Connection refused 2021-09-16T19:39:08.831 DEBUG:teuthology.orchestra.run.smithi118:> curl http://smithi118.front.sepia.ceph.com:8000 2021-09-16T19:39:08.851 INFO:teuthology.orchestra.run.smithi118.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-16T19:39:08.852 INFO:teuthology.orchestra.run.smithi118.stderr: Dload Upload Total Spent Left Speed 2021-09-16T19:39:08.853 INFO:teuthology.orchestra.run.smithi118.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2021-09-16T19:39:08.854 INFO:teuthology.orchestra.run.smithi118.stdout:anonymous 2021-09-16T19:39:08.854 INFO:tasks.rgw_multisite:creating zone.. 2021-09-16T19:39:08.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'K0MH58GXSJSVNR3I', '--secret', '93vysIyO7YxIyUlpTN7GyE2EQAEwRODi', '--endpoints', 'http://smithi109.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:39:08.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'K0MH58GXSJSVNR3I', '--secret', '93vysIyO7YxIyUlpTN7GyE2EQAEwRODi', '--endpoints', 'http://smithi109.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:39:08.856 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key K0MH58GXSJSVNR3I --secret 93vysIyO7YxIyUlpTN7GyE2EQAEwRODi --endpoints http://smithi109.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-16T19:39:08.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2021-09-16T19:39:08.905 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:39:08.906 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:39:08.931 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:39:08.930+0000 7f6e584a8380 0 failed reading obj info from .rgw.root:zone_info.ceba8a70-b881-4808-8acf-e8ddfa5f6daf: (2) No such file or directory 2021-09-16T19:39:08.932 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:39:08.930+0000 7f6e584a8380 0 WARNING: could not read zone params for zone id=ceba8a70-b881-4808-8acf-e8ddfa5f6daf name=test-zone2 2021-09-16T19:39:08.935 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:39:08.935 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "8df190b8-e36d-4ab9-bca1-30bba5bb5ec5", 2021-09-16T19:39:08.935 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zone3", 2021-09-16T19:39:08.936 INFO:teuthology.orchestra.run.smithi109.stdout: "domain_root": "test-zone3.rgw.meta:root", 2021-09-16T19:39:08.936 INFO:teuthology.orchestra.run.smithi109.stdout: "control_pool": "test-zone3.rgw.control", 2021-09-16T19:39:08.936 INFO:teuthology.orchestra.run.smithi109.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2021-09-16T19:39:08.936 INFO:teuthology.orchestra.run.smithi109.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2021-09-16T19:39:08.936 INFO:teuthology.orchestra.run.smithi109.stdout: "log_pool": "test-zone3.rgw.log", 2021-09-16T19:39:08.937 INFO:teuthology.orchestra.run.smithi109.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2021-09-16T19:39:08.937 INFO:teuthology.orchestra.run.smithi109.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2021-09-16T19:39:08.937 INFO:teuthology.orchestra.run.smithi109.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2021-09-16T19:39:08.937 INFO:teuthology.orchestra.run.smithi109.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2021-09-16T19:39:08.938 INFO:teuthology.orchestra.run.smithi109.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2021-09-16T19:39:08.938 INFO:teuthology.orchestra.run.smithi109.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2021-09-16T19:39:08.938 INFO:teuthology.orchestra.run.smithi109.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2021-09-16T19:39:08.938 INFO:teuthology.orchestra.run.smithi109.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2021-09-16T19:39:08.939 INFO:teuthology.orchestra.run.smithi109.stdout: "otp_pool": "test-zone3.rgw.otp", 2021-09-16T19:39:08.939 INFO:teuthology.orchestra.run.smithi109.stdout: "system_key": { 2021-09-16T19:39:08.940 INFO:teuthology.orchestra.run.smithi109.stdout: "access_key": "K0MH58GXSJSVNR3I", 2021-09-16T19:39:08.940 INFO:teuthology.orchestra.run.smithi109.stdout: "secret_key": "93vysIyO7YxIyUlpTN7GyE2EQAEwRODi" 2021-09-16T19:39:08.940 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:39:08.940 INFO:teuthology.orchestra.run.smithi109.stdout: "placement_pools": [ 2021-09-16T19:39:08.941 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:39:08.941 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "default-placement", 2021-09-16T19:39:08.941 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:39:08.941 INFO:teuthology.orchestra.run.smithi109.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2021-09-16T19:39:08.942 INFO:teuthology.orchestra.run.smithi109.stdout: "storage_classes": { 2021-09-16T19:39:08.942 INFO:teuthology.orchestra.run.smithi109.stdout: "STANDARD": { 2021-09-16T19:39:08.942 INFO:teuthology.orchestra.run.smithi109.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2021-09-16T19:39:08.942 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:39:08.942 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:39:08.943 INFO:teuthology.orchestra.run.smithi109.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2021-09-16T19:39:08.943 INFO:teuthology.orchestra.run.smithi109.stdout: "index_type": 0 2021-09-16T19:39:08.943 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:39:08.943 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:39:08.944 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:39:08.944 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T19:39:08.944 INFO:teuthology.orchestra.run.smithi109.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2021-09-16T19:39:08.944 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:39:08.946 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2021-09-16T19:39:09.994 INFO:teuthology.orchestra.run.smithi109.stderr:pool 'test-zone3.rgw.buckets.data' created 2021-09-16T19:39:10.010 DEBUG:teuthology.orchestra.run.smithi109:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2021-09-16T19:39:11.001 INFO:teuthology.orchestra.run.smithi109.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2021-09-16T19:39:11.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:39:11.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:39:11.025 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-16T19:39:11.070 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:39:11.071 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:39:11.093 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:39:11.094 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "8df190b8-e36d-4ab9-bca1-30bba5bb5ec5", 2021-09-16T19:39:11.094 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zone3", 2021-09-16T19:39:11.094 INFO:teuthology.orchestra.run.smithi109.stdout: "domain_root": "test-zone3.rgw.meta:root", 2021-09-16T19:39:11.094 INFO:teuthology.orchestra.run.smithi109.stdout: "control_pool": "test-zone3.rgw.control", 2021-09-16T19:39:11.095 INFO:teuthology.orchestra.run.smithi109.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2021-09-16T19:39:11.095 INFO:teuthology.orchestra.run.smithi109.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2021-09-16T19:39:11.095 INFO:teuthology.orchestra.run.smithi109.stdout: "log_pool": "test-zone3.rgw.log", 2021-09-16T19:39:11.095 INFO:teuthology.orchestra.run.smithi109.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2021-09-16T19:39:11.096 INFO:teuthology.orchestra.run.smithi109.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2021-09-16T19:39:11.096 INFO:teuthology.orchestra.run.smithi109.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2021-09-16T19:39:11.096 INFO:teuthology.orchestra.run.smithi109.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2021-09-16T19:39:11.096 INFO:teuthology.orchestra.run.smithi109.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2021-09-16T19:39:11.096 INFO:teuthology.orchestra.run.smithi109.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2021-09-16T19:39:11.097 INFO:teuthology.orchestra.run.smithi109.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2021-09-16T19:39:11.097 INFO:teuthology.orchestra.run.smithi109.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2021-09-16T19:39:11.097 INFO:teuthology.orchestra.run.smithi109.stdout: "otp_pool": "test-zone3.rgw.otp", 2021-09-16T19:39:11.097 INFO:teuthology.orchestra.run.smithi109.stdout: "system_key": { 2021-09-16T19:39:11.097 INFO:teuthology.orchestra.run.smithi109.stdout: "access_key": "K0MH58GXSJSVNR3I", 2021-09-16T19:39:11.098 INFO:teuthology.orchestra.run.smithi109.stdout: "secret_key": "93vysIyO7YxIyUlpTN7GyE2EQAEwRODi" 2021-09-16T19:39:11.098 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:39:11.098 INFO:teuthology.orchestra.run.smithi109.stdout: "placement_pools": [ 2021-09-16T19:39:11.098 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:39:11.099 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "default-placement", 2021-09-16T19:39:11.099 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:39:11.099 INFO:teuthology.orchestra.run.smithi109.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2021-09-16T19:39:11.099 INFO:teuthology.orchestra.run.smithi109.stdout: "storage_classes": { 2021-09-16T19:39:11.099 INFO:teuthology.orchestra.run.smithi109.stdout: "STANDARD": { 2021-09-16T19:39:11.100 INFO:teuthology.orchestra.run.smithi109.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2021-09-16T19:39:11.100 INFO:teuthology.orchestra.run.smithi109.stdout: "compression_type": "random" 2021-09-16T19:39:11.100 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:39:11.100 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:39:11.100 INFO:teuthology.orchestra.run.smithi109.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2021-09-16T19:39:11.101 INFO:teuthology.orchestra.run.smithi109.stdout: "index_type": 0 2021-09-16T19:39:11.101 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:39:11.101 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:39:11.101 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:39:11.102 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "", 2021-09-16T19:39:11.102 INFO:teuthology.orchestra.run.smithi109.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2021-09-16T19:39:11.102 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:39:11.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:39:11.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:39:11.104 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-16T19:39:11.144 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:39:11.145 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:39:11.153 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:39:11.152+0000 7fe706e31380 0 period (db0b3d90-59a0-4d0b-99ed-8408dd9cf54f does not have zone 8df190b8-e36d-4ab9-bca1-30bba5bb5ec5 configured 2021-09-16T19:39:19.055 INFO:teuthology.orchestra.run.smithi109.stderr:Sending period to new master zone 6395206f-e97e-464f-b7bd-c10485d78bf5 2021-09-16T19:39:19.083 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:39:19.084 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:39:19.084 INFO:teuthology.orchestra.run.smithi109.stdout: "epoch": 3, 2021-09-16T19:39:19.084 INFO:teuthology.orchestra.run.smithi109.stdout: "predecessor_uuid": "6be2ae27-983c-4bba-b75f-41d101865b26", 2021-09-16T19:39:19.084 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": [], 2021-09-16T19:39:19.084 INFO:teuthology.orchestra.run.smithi109.stdout: "period_map": { 2021-09-16T19:39:19.085 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:39:19.085 INFO:teuthology.orchestra.run.smithi109.stdout: "zonegroups": [ 2021-09-16T19:39:19.085 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:39:19.085 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "test-zonegroup", 2021-09-16T19:39:19.085 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zonegroup", 2021-09-16T19:39:19.086 INFO:teuthology.orchestra.run.smithi109.stdout: "api_name": "", 2021-09-16T19:39:19.086 INFO:teuthology.orchestra.run.smithi109.stdout: "is_master": "true", 2021-09-16T19:39:19.086 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [ 2021-09-16T19:39:19.086 INFO:teuthology.orchestra.run.smithi109.stdout: "http://smithi109.front.sepia.ceph.com:8000" 2021-09-16T19:39:19.087 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:39:19.087 INFO:teuthology.orchestra.run.smithi109.stdout: "hostnames": [], 2021-09-16T19:39:19.087 INFO:teuthology.orchestra.run.smithi109.stdout: "hostnames_s3website": [], 2021-09-16T19:39:19.087 INFO:teuthology.orchestra.run.smithi109.stdout: "master_zone": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:39:19.088 INFO:teuthology.orchestra.run.smithi109.stdout: "zones": [ 2021-09-16T19:39:19.088 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:39:19.088 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:39:19.088 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zone1", 2021-09-16T19:39:19.088 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [ 2021-09-16T19:39:19.088 INFO:teuthology.orchestra.run.smithi109.stdout: "http://smithi109.front.sepia.ceph.com:8000" 2021-09-16T19:39:19.089 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:39:19.089 INFO:teuthology.orchestra.run.smithi109.stdout: "log_meta": "false", 2021-09-16T19:39:19.089 INFO:teuthology.orchestra.run.smithi109.stdout: "log_data": "true", 2021-09-16T19:39:19.089 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_index_max_shards": 11, 2021-09-16T19:39:19.090 INFO:teuthology.orchestra.run.smithi109.stdout: "read_only": "false", 2021-09-16T19:39:19.090 INFO:teuthology.orchestra.run.smithi109.stdout: "tier_type": "", 2021-09-16T19:39:19.090 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from_all": "true", 2021-09-16T19:39:19.090 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from": [], 2021-09-16T19:39:19.090 INFO:teuthology.orchestra.run.smithi109.stdout: "redirect_zone": "" 2021-09-16T19:39:19.091 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:39:19.091 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:39:19.091 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "8df190b8-e36d-4ab9-bca1-30bba5bb5ec5", 2021-09-16T19:39:19.091 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zone3", 2021-09-16T19:39:19.091 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [ 2021-09-16T19:39:19.092 INFO:teuthology.orchestra.run.smithi109.stdout: "http://smithi109.front.sepia.ceph.com:8001" 2021-09-16T19:39:19.092 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:39:19.092 INFO:teuthology.orchestra.run.smithi109.stdout: "log_meta": "false", 2021-09-16T19:39:19.092 INFO:teuthology.orchestra.run.smithi109.stdout: "log_data": "true", 2021-09-16T19:39:19.092 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_index_max_shards": 11, 2021-09-16T19:39:19.093 INFO:teuthology.orchestra.run.smithi109.stdout: "read_only": "false", 2021-09-16T19:39:19.093 INFO:teuthology.orchestra.run.smithi109.stdout: "tier_type": "", 2021-09-16T19:39:19.093 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from_all": "true", 2021-09-16T19:39:19.093 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from": [], 2021-09-16T19:39:19.093 INFO:teuthology.orchestra.run.smithi109.stdout: "redirect_zone": "" 2021-09-16T19:39:19.094 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:39:19.094 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:39:19.094 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "ceba8a70-b881-4808-8acf-e8ddfa5f6daf", 2021-09-16T19:39:19.094 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zone2", 2021-09-16T19:39:19.095 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [ 2021-09-16T19:39:19.095 INFO:teuthology.orchestra.run.smithi109.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2021-09-16T19:39:19.095 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:39:19.095 INFO:teuthology.orchestra.run.smithi109.stdout: "log_meta": "false", 2021-09-16T19:39:19.096 INFO:teuthology.orchestra.run.smithi109.stdout: "log_data": "true", 2021-09-16T19:39:19.096 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_index_max_shards": 11, 2021-09-16T19:39:19.096 INFO:teuthology.orchestra.run.smithi109.stdout: "read_only": "false", 2021-09-16T19:39:19.096 INFO:teuthology.orchestra.run.smithi109.stdout: "tier_type": "", 2021-09-16T19:39:19.096 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from_all": "true", 2021-09-16T19:39:19.097 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from": [], 2021-09-16T19:39:19.097 INFO:teuthology.orchestra.run.smithi109.stdout: "redirect_zone": "" 2021-09-16T19:39:19.097 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:39:19.097 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:39:19.097 INFO:teuthology.orchestra.run.smithi109.stdout: "placement_targets": [ 2021-09-16T19:39:19.098 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:39:19.098 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "default-placement", 2021-09-16T19:39:19.098 INFO:teuthology.orchestra.run.smithi109.stdout: "tags": [], 2021-09-16T19:39:19.098 INFO:teuthology.orchestra.run.smithi109.stdout: "storage_classes": [ 2021-09-16T19:39:19.098 INFO:teuthology.orchestra.run.smithi109.stdout: "STANDARD" 2021-09-16T19:39:19.099 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:39:19.099 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:39:19.099 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:39:19.099 INFO:teuthology.orchestra.run.smithi109.stdout: "default_placement": "default-placement", 2021-09-16T19:39:19.099 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T19:39:19.100 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_policy": { 2021-09-16T19:39:19.100 INFO:teuthology.orchestra.run.smithi109.stdout: "groups": [] 2021-09-16T19:39:19.100 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:39:19.100 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:39:19.100 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:39:19.101 INFO:teuthology.orchestra.run.smithi109.stdout: "short_zone_ids": [ 2021-09-16T19:39:19.101 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:39:19.101 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:39:19.101 INFO:teuthology.orchestra.run.smithi109.stdout: "val": 3703685179 2021-09-16T19:39:19.101 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:39:19.102 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:39:19.102 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "8df190b8-e36d-4ab9-bca1-30bba5bb5ec5", 2021-09-16T19:39:19.102 INFO:teuthology.orchestra.run.smithi109.stdout: "val": 3204282247 2021-09-16T19:39:19.102 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:39:19.102 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:39:19.103 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "ceba8a70-b881-4808-8acf-e8ddfa5f6daf", 2021-09-16T19:39:19.103 INFO:teuthology.orchestra.run.smithi109.stdout: "val": 3287816121 2021-09-16T19:39:19.103 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:39:19.103 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:39:19.103 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:39:19.104 INFO:teuthology.orchestra.run.smithi109.stdout: "master_zonegroup": "test-zonegroup", 2021-09-16T19:39:19.104 INFO:teuthology.orchestra.run.smithi109.stdout: "master_zone": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:39:19.104 INFO:teuthology.orchestra.run.smithi109.stdout: "period_config": { 2021-09-16T19:39:19.104 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_quota": { 2021-09-16T19:39:19.105 INFO:teuthology.orchestra.run.smithi109.stdout: "enabled": false, 2021-09-16T19:39:19.105 INFO:teuthology.orchestra.run.smithi109.stdout: "check_on_raw": false, 2021-09-16T19:39:19.105 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size": -1, 2021-09-16T19:39:19.105 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size_kb": 0, 2021-09-16T19:39:19.105 INFO:teuthology.orchestra.run.smithi109.stdout: "max_objects": -1 2021-09-16T19:39:19.105 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:39:19.106 INFO:teuthology.orchestra.run.smithi109.stdout: "user_quota": { 2021-09-16T19:39:19.106 INFO:teuthology.orchestra.run.smithi109.stdout: "enabled": false, 2021-09-16T19:39:19.106 INFO:teuthology.orchestra.run.smithi109.stdout: "check_on_raw": false, 2021-09-16T19:39:19.107 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size": -1, 2021-09-16T19:39:19.107 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size_kb": 0, 2021-09-16T19:39:19.108 INFO:teuthology.orchestra.run.smithi109.stdout: "max_objects": -1 2021-09-16T19:39:19.108 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:39:19.108 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:39:19.108 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T19:39:19.109 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_name": "test-realm", 2021-09-16T19:39:19.109 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:39:19.109 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:39:19.110 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-09-16T19:39:19.110 INFO:teuthology.orchestra.run:waiting for 300 2021-09-16T19:39:19.148 INFO:tasks.rgw.c1.client.1.smithi109.stdout:2021-09-16T19:39:19.147+0000 7f9d9988e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 36597) UID: 0 2021-09-16T19:39:19.149 INFO:tasks.rgw.c1.client.1.smithi109.stdout:2021-09-16T19:39:19.147+0000 7f9daa731440 -1 shutting down 2021-09-16T19:39:21.971 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-09-16T19:39:25.212 INFO:tasks.rgw.c1.client.1:Stopped 2021-09-16T19:39:25.213 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-09-16T19:39:25.213 DEBUG:teuthology.orchestra.run.smithi109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-09-16T19:39:25.216 INFO:tasks.rgw.c1.client.1:Started 2021-09-16T19:39:25.216 DEBUG:teuthology.orchestra.run.smithi109:> curl http://smithi109.front.sepia.ceph.com:8001 2021-09-16T19:39:25.236 INFO:teuthology.orchestra.run.smithi109.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-16T19:39:25.236 INFO:teuthology.orchestra.run.smithi109.stderr: Dload Upload Total Spent Left Speed 2021-09-16T19:39:25.236 INFO:teuthology.orchestra.run.smithi109.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi109.front.sepia.ceph.com port 8001: Connection refused 2021-09-16T19:39:25.237 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-16T19:39:26.238 DEBUG:teuthology.orchestra.run.smithi109:> curl http://smithi109.front.sepia.ceph.com:8001 2021-09-16T19:39:26.258 INFO:teuthology.orchestra.run.smithi109.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-16T19:39:26.258 INFO:teuthology.orchestra.run.smithi109.stderr: Dload Upload Total Spent Left Speed 2021-09-16T19:39:26.259 INFO:teuthology.orchestra.run.smithi109.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2021-09-16T19:39:26.259 INFO:teuthology.orchestra.run.smithi109.stdout:anonymous 2021-09-16T19:39:26.260 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-09-16T19:39:26.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi109.front.sepia.ceph.com:8000', '--access-key', 'K0MH58GXSJSVNR3I', '--secret', '93vysIyO7YxIyUlpTN7GyE2EQAEwRODi', '--url', 'http://smithi109.front.sepia.ceph.com:8000', '--access-key', 'K0MH58GXSJSVNR3I', '--secret', '93vysIyO7YxIyUlpTN7GyE2EQAEwRODi', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:39:26.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi109.front.sepia.ceph.com:8000', '--access-key', 'K0MH58GXSJSVNR3I', '--secret', '93vysIyO7YxIyUlpTN7GyE2EQAEwRODi', '--url', 'http://smithi109.front.sepia.ceph.com:8000', '--access-key', 'K0MH58GXSJSVNR3I', '--secret', '93vysIyO7YxIyUlpTN7GyE2EQAEwRODi', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:39:26.261 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi109.front.sepia.ceph.com:8000 --access-key K0MH58GXSJSVNR3I --secret 93vysIyO7YxIyUlpTN7GyE2EQAEwRODi --url http://smithi109.front.sepia.ceph.com:8000 --access-key K0MH58GXSJSVNR3I --secret 93vysIyO7YxIyUlpTN7GyE2EQAEwRODi --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-16T19:39:26.318 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:39:26.318 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:39:26.348 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2021-09-16T19:39:26.347+0000 7f0de1d41700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.24db56c9-e102-4b9e-a672-6434941d0bb3.control err=-107 2021-09-16T19:39:26.356 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:39:26.356 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T19:39:26.357 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-realm", 2021-09-16T19:39:26.357 INFO:teuthology.orchestra.run.smithi118.stdout: "current_period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:39:26.357 INFO:teuthology.orchestra.run.smithi118.stdout: "epoch": 2 2021-09-16T19:39:26.357 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:39:26.359 INFO:tasks.rgw_multisite:creating zone.. 2021-09-16T19:39:26.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'K0MH58GXSJSVNR3I', '--secret', '93vysIyO7YxIyUlpTN7GyE2EQAEwRODi', '--endpoints', 'http://smithi118.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:39:26.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'K0MH58GXSJSVNR3I', '--secret', '93vysIyO7YxIyUlpTN7GyE2EQAEwRODi', '--endpoints', 'http://smithi118.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:39:26.360 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key K0MH58GXSJSVNR3I --secret 93vysIyO7YxIyUlpTN7GyE2EQAEwRODi --endpoints http://smithi118.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-16T19:39:26.414 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:39:26.414 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:39:26.436 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:39:26.435+0000 7f119e21f380 0 failed reading obj info from .rgw.root:zone_info.6395206f-e97e-464f-b7bd-c10485d78bf5: (2) No such file or directory 2021-09-16T19:39:26.436 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:39:26.435+0000 7f119e21f380 0 WARNING: could not read zone params for zone id=6395206f-e97e-464f-b7bd-c10485d78bf5 name=test-zone1 2021-09-16T19:39:26.436 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:39:26.436+0000 7f119e21f380 0 failed reading obj info from .rgw.root:zone_info.8df190b8-e36d-4ab9-bca1-30bba5bb5ec5: (2) No such file or directory 2021-09-16T19:39:26.437 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:39:26.436+0000 7f119e21f380 0 WARNING: could not read zone params for zone id=8df190b8-e36d-4ab9-bca1-30bba5bb5ec5 name=test-zone3 2021-09-16T19:39:26.440 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:39:26.440 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "c6342467-0ed0-40c7-a5ab-491a56113d8e", 2021-09-16T19:39:26.440 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zone4", 2021-09-16T19:39:26.440 INFO:teuthology.orchestra.run.smithi118.stdout: "domain_root": "test-zone4.rgw.meta:root", 2021-09-16T19:39:26.441 INFO:teuthology.orchestra.run.smithi118.stdout: "control_pool": "test-zone4.rgw.control", 2021-09-16T19:39:26.441 INFO:teuthology.orchestra.run.smithi118.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2021-09-16T19:39:26.441 INFO:teuthology.orchestra.run.smithi118.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2021-09-16T19:39:26.441 INFO:teuthology.orchestra.run.smithi118.stdout: "log_pool": "test-zone4.rgw.log", 2021-09-16T19:39:26.442 INFO:teuthology.orchestra.run.smithi118.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2021-09-16T19:39:26.442 INFO:teuthology.orchestra.run.smithi118.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2021-09-16T19:39:26.442 INFO:teuthology.orchestra.run.smithi118.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2021-09-16T19:39:26.442 INFO:teuthology.orchestra.run.smithi118.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2021-09-16T19:39:26.442 INFO:teuthology.orchestra.run.smithi118.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2021-09-16T19:39:26.443 INFO:teuthology.orchestra.run.smithi118.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2021-09-16T19:39:26.443 INFO:teuthology.orchestra.run.smithi118.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2021-09-16T19:39:26.443 INFO:teuthology.orchestra.run.smithi118.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2021-09-16T19:39:26.443 INFO:teuthology.orchestra.run.smithi118.stdout: "otp_pool": "test-zone4.rgw.otp", 2021-09-16T19:39:26.444 INFO:teuthology.orchestra.run.smithi118.stdout: "system_key": { 2021-09-16T19:39:26.444 INFO:teuthology.orchestra.run.smithi118.stdout: "access_key": "K0MH58GXSJSVNR3I", 2021-09-16T19:39:26.444 INFO:teuthology.orchestra.run.smithi118.stdout: "secret_key": "93vysIyO7YxIyUlpTN7GyE2EQAEwRODi" 2021-09-16T19:39:26.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:26.445 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_pools": [ 2021-09-16T19:39:26.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:26.445 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "default-placement", 2021-09-16T19:39:26.445 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:26.446 INFO:teuthology.orchestra.run.smithi118.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2021-09-16T19:39:26.446 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": { 2021-09-16T19:39:26.446 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD": { 2021-09-16T19:39:26.446 INFO:teuthology.orchestra.run.smithi118.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2021-09-16T19:39:26.447 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:26.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:26.447 INFO:teuthology.orchestra.run.smithi118.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2021-09-16T19:39:26.447 INFO:teuthology.orchestra.run.smithi118.stdout: "index_type": 0 2021-09-16T19:39:26.447 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:26.448 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:26.448 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:39:26.448 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_config": { 2021-09-16T19:39:26.448 INFO:teuthology.orchestra.run.smithi118.stdout: "event_retention_days": 7, 2021-09-16T19:39:26.449 INFO:teuthology.orchestra.run.smithi118.stdout: "start_with_full_sync": false 2021-09-16T19:39:26.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:26.449 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T19:39:26.449 INFO:teuthology.orchestra.run.smithi118.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2021-09-16T19:39:26.449 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:39:26.451 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2021-09-16T19:39:27.613 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'test-zone4.rgw.buckets.data' created 2021-09-16T19:39:27.631 DEBUG:teuthology.orchestra.run.smithi118:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2021-09-16T19:39:27.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-09-16T19:39:28.640 INFO:teuthology.orchestra.run.smithi118.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2021-09-16T19:39:28.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:39:28.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:39:28.660 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-16T19:39:28.707 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:39:28.707 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:39:28.731 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:39:28.731 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "c6342467-0ed0-40c7-a5ab-491a56113d8e", 2021-09-16T19:39:28.732 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zone4", 2021-09-16T19:39:28.732 INFO:teuthology.orchestra.run.smithi118.stdout: "domain_root": "test-zone4.rgw.meta:root", 2021-09-16T19:39:28.732 INFO:teuthology.orchestra.run.smithi118.stdout: "control_pool": "test-zone4.rgw.control", 2021-09-16T19:39:28.732 INFO:teuthology.orchestra.run.smithi118.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2021-09-16T19:39:28.732 INFO:teuthology.orchestra.run.smithi118.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2021-09-16T19:39:28.733 INFO:teuthology.orchestra.run.smithi118.stdout: "log_pool": "test-zone4.rgw.log", 2021-09-16T19:39:28.733 INFO:teuthology.orchestra.run.smithi118.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2021-09-16T19:39:28.733 INFO:teuthology.orchestra.run.smithi118.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2021-09-16T19:39:28.733 INFO:teuthology.orchestra.run.smithi118.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2021-09-16T19:39:28.734 INFO:teuthology.orchestra.run.smithi118.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2021-09-16T19:39:28.734 INFO:teuthology.orchestra.run.smithi118.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2021-09-16T19:39:28.734 INFO:teuthology.orchestra.run.smithi118.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2021-09-16T19:39:28.734 INFO:teuthology.orchestra.run.smithi118.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2021-09-16T19:39:28.734 INFO:teuthology.orchestra.run.smithi118.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2021-09-16T19:39:28.735 INFO:teuthology.orchestra.run.smithi118.stdout: "otp_pool": "test-zone4.rgw.otp", 2021-09-16T19:39:28.735 INFO:teuthology.orchestra.run.smithi118.stdout: "system_key": { 2021-09-16T19:39:28.735 INFO:teuthology.orchestra.run.smithi118.stdout: "access_key": "K0MH58GXSJSVNR3I", 2021-09-16T19:39:28.735 INFO:teuthology.orchestra.run.smithi118.stdout: "secret_key": "93vysIyO7YxIyUlpTN7GyE2EQAEwRODi" 2021-09-16T19:39:28.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:28.735 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_pools": [ 2021-09-16T19:39:28.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:28.736 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "default-placement", 2021-09-16T19:39:28.736 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:28.736 INFO:teuthology.orchestra.run.smithi118.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2021-09-16T19:39:28.736 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": { 2021-09-16T19:39:28.737 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD": { 2021-09-16T19:39:28.737 INFO:teuthology.orchestra.run.smithi118.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2021-09-16T19:39:28.737 INFO:teuthology.orchestra.run.smithi118.stdout: "compression_type": "random" 2021-09-16T19:39:28.737 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:28.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:28.738 INFO:teuthology.orchestra.run.smithi118.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2021-09-16T19:39:28.738 INFO:teuthology.orchestra.run.smithi118.stdout: "index_type": 0 2021-09-16T19:39:28.738 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:28.738 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:28.738 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:39:28.739 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_config": { 2021-09-16T19:39:28.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event_retention_days": 7, 2021-09-16T19:39:28.739 INFO:teuthology.orchestra.run.smithi118.stdout: "start_with_full_sync": false 2021-09-16T19:39:28.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:28.739 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "", 2021-09-16T19:39:28.740 INFO:teuthology.orchestra.run.smithi118.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2021-09-16T19:39:28.740 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:39:28.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:39:28.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:39:28.742 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-16T19:39:28.783 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:39:28.783 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:39:28.791 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:39:28.791+0000 7ff0f35c4380 0 period (db0b3d90-59a0-4d0b-99ed-8408dd9cf54f does not have zone c6342467-0ed0-40c7-a5ab-491a56113d8e configured 2021-09-16T19:39:36.696 INFO:teuthology.orchestra.run.smithi118.stderr:Sending period to new master zone 6395206f-e97e-464f-b7bd-c10485d78bf5 2021-09-16T19:39:36.721 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:39:36.722 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:39:36.722 INFO:teuthology.orchestra.run.smithi118.stdout: "epoch": 4, 2021-09-16T19:39:36.722 INFO:teuthology.orchestra.run.smithi118.stdout: "predecessor_uuid": "6be2ae27-983c-4bba-b75f-41d101865b26", 2021-09-16T19:39:36.723 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": [], 2021-09-16T19:39:36.723 INFO:teuthology.orchestra.run.smithi118.stdout: "period_map": { 2021-09-16T19:39:36.723 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:39:36.723 INFO:teuthology.orchestra.run.smithi118.stdout: "zonegroups": [ 2021-09-16T19:39:36.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:36.724 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "test-zonegroup", 2021-09-16T19:39:36.724 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zonegroup", 2021-09-16T19:39:36.724 INFO:teuthology.orchestra.run.smithi118.stdout: "api_name": "", 2021-09-16T19:39:36.724 INFO:teuthology.orchestra.run.smithi118.stdout: "is_master": "true", 2021-09-16T19:39:36.725 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2021-09-16T19:39:36.725 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi109.front.sepia.ceph.com:8000" 2021-09-16T19:39:36.725 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:39:36.725 INFO:teuthology.orchestra.run.smithi118.stdout: "hostnames": [], 2021-09-16T19:39:36.726 INFO:teuthology.orchestra.run.smithi118.stdout: "hostnames_s3website": [], 2021-09-16T19:39:36.726 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zone": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:39:36.726 INFO:teuthology.orchestra.run.smithi118.stdout: "zones": [ 2021-09-16T19:39:36.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:36.727 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:39:36.727 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zone1", 2021-09-16T19:39:36.727 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2021-09-16T19:39:36.727 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi109.front.sepia.ceph.com:8000" 2021-09-16T19:39:36.728 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:39:36.728 INFO:teuthology.orchestra.run.smithi118.stdout: "log_meta": "false", 2021-09-16T19:39:36.728 INFO:teuthology.orchestra.run.smithi118.stdout: "log_data": "true", 2021-09-16T19:39:36.728 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_index_max_shards": 11, 2021-09-16T19:39:36.728 INFO:teuthology.orchestra.run.smithi118.stdout: "read_only": "false", 2021-09-16T19:39:36.729 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_type": "", 2021-09-16T19:39:36.729 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from_all": "true", 2021-09-16T19:39:36.729 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from": [], 2021-09-16T19:39:36.730 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_zone": "" 2021-09-16T19:39:36.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:36.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:36.730 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "8df190b8-e36d-4ab9-bca1-30bba5bb5ec5", 2021-09-16T19:39:36.731 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zone3", 2021-09-16T19:39:36.731 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2021-09-16T19:39:36.731 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi109.front.sepia.ceph.com:8001" 2021-09-16T19:39:36.731 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:39:36.732 INFO:teuthology.orchestra.run.smithi118.stdout: "log_meta": "false", 2021-09-16T19:39:36.732 INFO:teuthology.orchestra.run.smithi118.stdout: "log_data": "true", 2021-09-16T19:39:36.732 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_index_max_shards": 11, 2021-09-16T19:39:36.732 INFO:teuthology.orchestra.run.smithi118.stdout: "read_only": "false", 2021-09-16T19:39:36.732 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_type": "", 2021-09-16T19:39:36.733 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from_all": "true", 2021-09-16T19:39:36.733 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from": [], 2021-09-16T19:39:36.733 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_zone": "" 2021-09-16T19:39:36.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:36.734 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:36.734 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "c6342467-0ed0-40c7-a5ab-491a56113d8e", 2021-09-16T19:39:36.734 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zone4", 2021-09-16T19:39:36.734 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2021-09-16T19:39:36.734 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2021-09-16T19:39:36.735 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:39:36.735 INFO:teuthology.orchestra.run.smithi118.stdout: "log_meta": "false", 2021-09-16T19:39:36.735 INFO:teuthology.orchestra.run.smithi118.stdout: "log_data": "true", 2021-09-16T19:39:36.735 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_index_max_shards": 11, 2021-09-16T19:39:36.736 INFO:teuthology.orchestra.run.smithi118.stdout: "read_only": "false", 2021-09-16T19:39:36.736 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_type": "pubsub", 2021-09-16T19:39:36.736 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from_all": "false", 2021-09-16T19:39:36.736 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from": [ 2021-09-16T19:39:36.737 INFO:teuthology.orchestra.run.smithi118.stdout: "test-zone1" 2021-09-16T19:39:36.737 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:39:36.737 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_zone": "" 2021-09-16T19:39:36.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:36.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:36.738 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "ceba8a70-b881-4808-8acf-e8ddfa5f6daf", 2021-09-16T19:39:36.738 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zone2", 2021-09-16T19:39:36.738 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2021-09-16T19:39:36.739 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2021-09-16T19:39:36.739 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:39:36.739 INFO:teuthology.orchestra.run.smithi118.stdout: "log_meta": "false", 2021-09-16T19:39:36.739 INFO:teuthology.orchestra.run.smithi118.stdout: "log_data": "true", 2021-09-16T19:39:36.740 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_index_max_shards": 11, 2021-09-16T19:39:36.740 INFO:teuthology.orchestra.run.smithi118.stdout: "read_only": "false", 2021-09-16T19:39:36.740 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_type": "", 2021-09-16T19:39:36.740 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from_all": "true", 2021-09-16T19:39:36.741 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from": [], 2021-09-16T19:39:36.741 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_zone": "" 2021-09-16T19:39:36.741 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:36.741 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:39:36.741 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_targets": [ 2021-09-16T19:39:36.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:36.742 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default-placement", 2021-09-16T19:39:36.742 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2021-09-16T19:39:36.742 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": [ 2021-09-16T19:39:36.742 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD" 2021-09-16T19:39:36.743 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:39:36.743 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:36.743 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:39:36.743 INFO:teuthology.orchestra.run.smithi118.stdout: "default_placement": "default-placement", 2021-09-16T19:39:36.744 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T19:39:36.744 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_policy": { 2021-09-16T19:39:36.744 INFO:teuthology.orchestra.run.smithi118.stdout: "groups": [] 2021-09-16T19:39:36.744 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:36.744 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:36.745 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T19:39:36.745 INFO:teuthology.orchestra.run.smithi118.stdout: "short_zone_ids": [ 2021-09-16T19:39:36.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:36.747 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:39:36.747 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 3703685179 2021-09-16T19:39:36.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:36.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:36.748 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "8df190b8-e36d-4ab9-bca1-30bba5bb5ec5", 2021-09-16T19:39:36.748 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 3204282247 2021-09-16T19:39:36.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:36.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:36.748 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "c6342467-0ed0-40c7-a5ab-491a56113d8e", 2021-09-16T19:39:36.749 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 1461382917 2021-09-16T19:39:36.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:36.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:36.749 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "ceba8a70-b881-4808-8acf-e8ddfa5f6daf", 2021-09-16T19:39:36.750 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 3287816121 2021-09-16T19:39:36.750 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:36.750 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:39:36.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:36.750 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zonegroup": "test-zonegroup", 2021-09-16T19:39:36.751 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zone": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T19:39:36.751 INFO:teuthology.orchestra.run.smithi118.stdout: "period_config": { 2021-09-16T19:39:36.751 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_quota": { 2021-09-16T19:39:36.751 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": false, 2021-09-16T19:39:36.752 INFO:teuthology.orchestra.run.smithi118.stdout: "check_on_raw": false, 2021-09-16T19:39:36.752 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size": -1, 2021-09-16T19:39:36.752 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size_kb": 0, 2021-09-16T19:39:36.752 INFO:teuthology.orchestra.run.smithi118.stdout: "max_objects": -1 2021-09-16T19:39:36.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:36.753 INFO:teuthology.orchestra.run.smithi118.stdout: "user_quota": { 2021-09-16T19:39:36.753 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": false, 2021-09-16T19:39:36.753 INFO:teuthology.orchestra.run.smithi118.stdout: "check_on_raw": false, 2021-09-16T19:39:36.753 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size": -1, 2021-09-16T19:39:36.754 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size_kb": 0, 2021-09-16T19:39:36.754 INFO:teuthology.orchestra.run.smithi118.stdout: "max_objects": -1 2021-09-16T19:39:36.754 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:36.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:36.755 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T19:39:36.755 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_name": "test-realm", 2021-09-16T19:39:36.755 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:36.755 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:39:36.756 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-09-16T19:39:36.756 INFO:teuthology.orchestra.run:waiting for 300 2021-09-16T19:39:36.789 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T19:39:36.787+0000 7f02fbd5b700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 36436) UID: 0 2021-09-16T19:39:36.789 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T19:39:36.787+0000 7f030cbfe440 -1 shutting down 2021-09-16T19:39:37.311 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-09-16T19:39:42.858 INFO:tasks.rgw.c2.client.1:Stopped 2021-09-16T19:39:42.859 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-09-16T19:39:42.859 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-09-16T19:39:42.862 INFO:tasks.rgw.c2.client.1:Started 2021-09-16T19:39:42.863 DEBUG:teuthology.orchestra.run.smithi118:> curl http://smithi118.front.sepia.ceph.com:8001 2021-09-16T19:39:42.883 INFO:teuthology.orchestra.run.smithi118.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-16T19:39:42.883 INFO:teuthology.orchestra.run.smithi118.stderr: Dload Upload Total Spent Left Speed 2021-09-16T19:39:42.884 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-16T19:39:42.885 INFO:teuthology.orchestra.run.smithi118.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi118.front.sepia.ceph.com port 8001: Connection refused 2021-09-16T19:39:42.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~6s 2021-09-16T19:39:43.886 DEBUG:teuthology.orchestra.run.smithi118:> curl http://smithi118.front.sepia.ceph.com:8001 2021-09-16T19:39:43.906 INFO:teuthology.orchestra.run.smithi118.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-16T19:39:43.906 INFO:teuthology.orchestra.run.smithi118.stderr: Dload Upload Total Spent Left Speed 2021-09-16T19:39:43.907 INFO:teuthology.orchestra.run.smithi118.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 130k 0 --:--:-- --:--:-- --:--:-- 130k 2021-09-16T19:39:43.908 INFO:teuthology.orchestra.run.smithi118.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-0061439d7f-10e7-test-zone4","HostId":"10e7-test-zone4-test-zonegroup"} 2021-09-16T19:39:43.908 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2021-09-16T19:39:43.908 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2021-09-16T19:39:43.924 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph-c_4e2e64ee99b6cbb34351964aad0f9df4874dea6d/qa/../src/test/rgw/rgw_multi 2021-09-16T19:39:43.939 INFO:tasks.rgw_multisite_tests:creating test user.. 2021-09-16T19:39:43.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:39:43.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T19:39:43.940 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-16T19:39:43.994 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:39:43.994 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:39:44.048 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:39:44.048 INFO:teuthology.orchestra.run.smithi109.stdout: "user_id": "rgw-multisite-test-user", 2021-09-16T19:39:44.048 INFO:teuthology.orchestra.run.smithi109.stdout: "display_name": "Multisite Test User", 2021-09-16T19:39:44.049 INFO:teuthology.orchestra.run.smithi109.stdout: "email": "", 2021-09-16T19:39:44.049 INFO:teuthology.orchestra.run.smithi109.stdout: "suspended": 0, 2021-09-16T19:39:44.049 INFO:teuthology.orchestra.run.smithi109.stdout: "max_buckets": 1000, 2021-09-16T19:39:44.049 INFO:teuthology.orchestra.run.smithi109.stdout: "subusers": [], 2021-09-16T19:39:44.050 INFO:teuthology.orchestra.run.smithi109.stdout: "keys": [ 2021-09-16T19:39:44.050 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:39:44.050 INFO:teuthology.orchestra.run.smithi109.stdout: "user": "rgw-multisite-test-user", 2021-09-16T19:39:44.050 INFO:teuthology.orchestra.run.smithi109.stdout: "access_key": "799OENMRSPIX6MOUBTSB", 2021-09-16T19:39:44.050 INFO:teuthology.orchestra.run.smithi109.stdout: "secret_key": "O8mLREJPwXfMrZ1cjBx10KSkcbq5SnH17iBPm5Xh" 2021-09-16T19:39:44.051 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:39:44.051 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T19:39:44.051 INFO:teuthology.orchestra.run.smithi109.stdout: "swift_keys": [], 2021-09-16T19:39:44.051 INFO:teuthology.orchestra.run.smithi109.stdout: "caps": [], 2021-09-16T19:39:44.051 INFO:teuthology.orchestra.run.smithi109.stdout: "op_mask": "read, write, delete", 2021-09-16T19:39:44.052 INFO:teuthology.orchestra.run.smithi109.stdout: "default_placement": "", 2021-09-16T19:39:44.052 INFO:teuthology.orchestra.run.smithi109.stdout: "default_storage_class": "", 2021-09-16T19:39:44.052 INFO:teuthology.orchestra.run.smithi109.stdout: "placement_tags": [], 2021-09-16T19:39:44.052 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_quota": { 2021-09-16T19:39:44.052 INFO:teuthology.orchestra.run.smithi109.stdout: "enabled": false, 2021-09-16T19:39:44.053 INFO:teuthology.orchestra.run.smithi109.stdout: "check_on_raw": false, 2021-09-16T19:39:44.053 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size": -1, 2021-09-16T19:39:44.053 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size_kb": 0, 2021-09-16T19:39:44.053 INFO:teuthology.orchestra.run.smithi109.stdout: "max_objects": -1 2021-09-16T19:39:44.053 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:39:44.054 INFO:teuthology.orchestra.run.smithi109.stdout: "user_quota": { 2021-09-16T19:39:44.054 INFO:teuthology.orchestra.run.smithi109.stdout: "enabled": false, 2021-09-16T19:39:44.054 INFO:teuthology.orchestra.run.smithi109.stdout: "check_on_raw": false, 2021-09-16T19:39:44.054 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size": -1, 2021-09-16T19:39:44.055 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size_kb": 0, 2021-09-16T19:39:44.055 INFO:teuthology.orchestra.run.smithi109.stdout: "max_objects": -1 2021-09-16T19:39:44.055 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:39:44.056 INFO:teuthology.orchestra.run.smithi109.stdout: "temp_url_keys": [], 2021-09-16T19:39:44.056 INFO:teuthology.orchestra.run.smithi109.stdout: "type": "rgw", 2021-09-16T19:39:44.056 INFO:teuthology.orchestra.run.smithi109.stdout: "mfa_ids": [] 2021-09-16T19:39:44.056 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:39:44.056 INFO:teuthology.orchestra.run.smithi109.stdout: 2021-09-16T19:39:44.060 INFO:rgw_multi.tests:meta checkpoint 2021-09-16T19:39:44.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:39:44.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:39:44.061 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:39:44.102 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:39:44.102 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:39:44.142 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:39:44.154 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:39:44.154 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:39:44.154 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:39:44.154 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:39:44.155 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:39:44.155 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:39:44.155 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:39:44.155 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:39:44.156 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:39:44.156 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:39:44.156 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:39:44.156 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:39:44.157 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:39:44.157 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:39:44.157 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:39:44.157 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:39:44.158 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:39:44.158 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:39:44.159 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:39:44.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:39:44.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:39:44.160 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:39:44.205 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:39:44.206 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:39:44.254 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:39:44.254 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:39:44.255 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:39:44.255 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:39:44.255 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:39:44.256 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:39:44.256 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:44.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:44.256 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:39:44.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:44.257 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:39:44.257 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:44.257 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:39:44.257 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:39:44.258 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:39:44.258 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:39:44.258 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:39:44.258 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:39:44.259 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:44.259 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:44.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:44.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:44.260 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:39:44.260 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:44.260 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:39:44.260 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:39:44.260 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:39:44.261 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:39:44.261 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:39:44.261 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:39:44.261 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:44.261 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:44.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:44.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:44.262 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:39:44.262 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:44.263 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:39:44.263 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:39:44.263 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:39:44.263 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:39:44.264 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:39:44.264 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:39:44.264 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:44.264 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:44.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:44.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:44.265 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:39:44.265 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:44.266 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:39:44.266 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:39:44.266 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:39:44.266 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:39:44.267 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:39:44.267 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:39:44.267 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:44.267 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:44.268 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:44.268 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:39:44.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:44.268 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:39:44.269 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:39:44.269 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:39:44.269 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:44.269 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:39:44.271 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:39:48.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2021-09-16T19:39:49.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:39:49.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:39:49.273 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:39:49.320 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:39:49.321 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:39:49.370 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:39:49.371 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:39:49.371 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:39:49.371 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:39:49.371 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:39:49.372 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:39:49.372 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:49.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:49.372 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:39:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:39:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:39:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:39:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:39:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:39:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:39:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:39:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:39:49.376 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:49.376 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:39:49.376 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:39:49.376 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:39:49.376 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:39:49.377 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:39:49.377 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:39:49.377 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:49.377 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:49.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:49.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:49.378 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:39:49.378 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:49.379 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:39:49.379 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:39:49.379 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:39:49.379 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:39:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:39:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:39:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:49.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:49.381 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:39:49.381 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:49.381 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:39:49.381 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:39:49.382 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:39:49.382 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:39:49.382 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:39:49.382 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:39:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:39:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:49.384 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:39:49.384 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:39:49.384 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:39:49.384 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:49.384 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:39:49.385 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:39:54.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:39:54.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:39:54.387 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:39:54.429 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:39:54.430 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:39:54.476 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:39:54.477 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:39:54.477 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:39:54.477 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:39:54.477 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:39:54.477 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:39:54.478 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:54.478 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:54.478 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:39:54.478 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:54.479 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:39:54.479 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:54.479 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:39:54.479 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:39:54.479 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:39:54.480 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:39:54.480 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:39:54.480 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:39:54.480 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:54.481 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:54.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:54.481 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:54.481 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:39:54.481 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:54.482 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:39:54.482 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:39:54.482 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:39:54.482 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:39:54.483 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:39:54.483 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:39:54.483 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:54.483 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:54.484 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:54.484 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:54.484 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:39:54.484 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:54.484 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:39:54.485 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:39:54.485 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:39:54.485 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:39:54.485 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:39:54.486 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:39:54.486 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:54.486 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:54.486 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:54.486 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:54.487 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:39:54.487 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:54.487 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:39:54.487 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:39:54.488 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:39:54.488 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:39:54.488 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:39:54.488 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:39:54.488 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:54.489 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:54.489 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:54.489 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:39:54.489 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:54.489 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:39:54.490 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:39:54.490 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:39:54.490 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:54.490 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:39:54.492 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:39:59.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:39:59.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:39:59.494 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:39:59.543 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:39:59.544 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:39:59.595 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:39:59.595 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:39:59.596 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:39:59.596 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:39:59.596 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:39:59.596 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:39:59.597 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:59.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:59.597 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:39:59.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:59.598 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:39:59.598 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:59.598 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:39:59.599 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:39:59.599 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:39:59.599 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:39:59.599 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:39:59.600 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:39:59.600 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:59.600 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:59.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:59.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:59.601 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:39:59.601 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:59.601 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:39:59.601 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:39:59.602 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:39:59.602 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:39:59.602 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:39:59.602 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:39:59.603 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:59.603 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:59.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:59.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:59.603 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:39:59.604 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:59.604 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:39:59.604 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:39:59.604 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:39:59.604 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:39:59.605 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:39:59.605 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:39:59.605 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:59.605 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:59.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:59.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:39:59.606 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:39:59.606 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:39:59.606 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:39:59.607 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:39:59.607 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:39:59.607 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:39:59.608 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:39:59.608 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:39:59.608 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:39:59.609 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:59.609 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:59.609 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:39:59.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:39:59.609 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:39:59.610 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:39:59.610 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:39:59.610 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:39:59.611 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:39:59.612 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:40:04.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:04.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:04.615 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:40:04.664 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:40:04.664 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:40:04.715 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:40:04.715 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:40:04.715 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:40:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:40:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:40:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:40:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:04.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:04.717 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:40:04.717 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:04.717 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:40:04.717 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:40:04.718 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:04.719 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:04.719 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:04.719 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:04.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:40:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:04.720 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:04.721 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:04.722 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:40:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:04.723 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:04.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:04.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:04.725 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:40:04.725 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:04.725 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:04.725 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:04.726 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:40:04.727 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:04.728 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:40:04.728 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:40:04.728 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:40:04.728 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:04.728 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:40:04.729 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:40:09.730 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:09.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:09.731 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:40:09.780 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:40:09.780 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:40:09.836 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:40:09.836 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:40:09.837 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:40:09.837 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:40:09.837 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:40:09.837 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:40:09.837 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:09.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:09.838 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:40:09.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:09.839 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:40:09.839 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:09.839 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:09.839 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:09.840 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:09.840 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:40:09.840 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:09.840 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:09.840 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:09.841 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:09.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:09.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:09.841 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:40:09.842 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:09.842 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:09.842 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:09.842 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:09.843 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:09.843 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:09.843 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:09.843 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:09.843 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:09.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:09.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:09.844 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:40:09.844 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:09.845 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:09.845 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:09.845 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:09.845 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:09.845 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:09.846 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:09.846 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:09.846 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:09.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:09.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:09.847 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:40:09.847 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:09.847 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:09.848 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:09.848 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:09.848 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:09.848 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:09.848 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:09.849 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:09.849 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:09.849 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:09.849 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:40:09.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:09.850 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:40:09.850 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:40:09.850 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:40:09.850 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:09.851 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:40:09.852 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:40:14.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:14.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:14.854 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:40:14.901 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:40:14.902 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:40:14.951 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:40:14.952 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:40:14.952 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:40:14.952 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:40:14.952 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:40:14.953 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:40:14.953 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:14.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:14.953 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:40:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:40:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:40:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:40:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:40:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:40:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:40:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:40:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:40:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:40:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:14.966 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:40:14.967 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:40:19.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:19.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:19.970 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:40:20.019 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:40:20.019 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:40:20.075 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:40:20.076 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:40:20.076 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:40:20.076 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:40:20.077 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:40:20.077 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:40:20.077 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:20.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:20.078 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:40:20.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:20.078 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:40:20.078 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:20.078 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:20.079 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:20.079 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:20.079 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:40:20.079 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:20.080 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:20.080 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:20.080 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:20.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:20.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:20.081 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:40:20.081 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:20.081 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:20.081 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:20.082 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:20.082 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:20.082 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:20.082 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:20.082 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:20.083 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:20.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:20.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:20.083 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:40:20.084 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:20.084 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:20.084 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:20.084 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:20.085 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:20.085 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:20.085 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:20.085 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:20.085 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:20.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:20.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:20.086 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:40:20.086 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:20.086 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:20.087 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:20.087 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:20.087 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:20.087 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:20.088 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:20.088 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:20.088 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:20.088 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:20.088 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:40:20.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:20.089 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:40:20.089 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:40:20.089 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:40:20.090 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:20.090 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:40:20.092 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:40:25.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:25.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:25.095 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:40:25.143 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:40:25.143 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:40:25.192 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:40:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:40:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:40:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:40:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:40:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:40:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:40:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:40:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:40:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:40:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:40:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:40:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:40:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:40:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:40:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:40:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:25.209 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:40:25.211 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:40:30.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:30.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:30.213 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:40:30.262 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:40:30.262 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:40:30.319 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:40:30.319 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:40:30.320 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:40:30.320 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:40:30.320 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:40:30.320 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:40:30.320 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:30.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:30.321 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:40:30.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:30.321 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:40:30.322 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:30.322 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:30.322 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:30.322 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:30.322 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:40:30.323 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:30.323 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:30.323 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:30.323 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:30.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:30.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:30.324 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:40:30.324 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:30.324 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:30.325 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:30.325 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:30.325 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:30.325 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:30.326 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:30.326 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:30.326 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:30.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:30.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:30.327 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:40:30.327 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:30.327 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:30.327 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:30.328 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:30.328 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:30.328 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:30.328 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:30.328 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:30.329 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:30.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:30.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:30.329 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:40:30.329 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:30.330 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:30.330 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:30.330 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:30.330 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:30.331 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:30.331 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:30.331 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:30.331 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:30.331 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:30.332 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:40:30.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:30.332 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:40:30.332 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:40:30.333 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:40:30.333 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:30.333 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:40:30.334 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:40:35.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:35.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:35.337 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:40:35.385 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:40:35.385 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:40:35.435 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:40:35.436 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:40:35.436 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:40:35.436 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:40:35.436 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:40:35.437 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:40:35.437 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:35.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:35.437 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:40:35.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:35.438 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:40:35.438 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:35.438 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:35.438 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:35.439 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:35.439 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:40:35.439 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:35.439 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:35.440 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:35.440 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:35.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:35.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:35.440 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:40:35.441 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:35.441 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:35.441 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:35.441 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:35.441 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:35.442 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:35.442 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:35.442 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:35.442 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:35.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:35.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:35.443 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:40:35.443 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:35.443 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:35.444 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:35.444 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:35.444 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:35.444 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:35.445 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:35.445 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:35.445 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:35.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:35.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:35.446 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:40:35.446 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:35.446 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:35.446 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:35.446 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:35.447 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:35.447 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:35.447 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:35.447 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:35.448 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:35.448 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:35.448 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:40:35.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:35.448 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:40:35.449 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:40:35.449 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:40:35.449 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:35.449 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:40:35.450 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:40:40.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:40.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:40.453 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:40:40.501 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:40:40.501 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:40:40.554 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:40:40.555 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:40:40.555 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:40:40.555 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:40:40.555 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:40:40.556 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:40:40.556 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:40.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:40.556 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:40:40.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:40.557 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:40:40.557 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:40.557 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:40.557 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:40.558 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:40.558 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:40:40.558 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:40.558 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:40.559 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:40.559 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:40.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:40.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:40.560 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:40:40.560 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:40.560 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:40.560 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:40.561 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:40.561 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:40.561 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:40.561 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:40.562 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:40.562 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:40.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:40.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:40.562 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:40:40.563 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:40.563 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:40.563 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:40.563 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:40.563 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:40.564 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:40.564 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:40.564 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:40.564 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:40.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:40.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:40.565 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:40:40.565 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:40.565 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:40.566 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:40.566 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:40.566 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:40.566 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:40.567 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:40.567 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:40.567 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:40.567 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:40.567 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:40:40.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:40.568 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:40:40.568 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:40:40.568 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:40:40.568 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:40.569 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:40:40.570 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:40:45.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:45.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:45.573 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:40:45.623 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:40:45.624 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:40:45.675 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:40:45.675 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:40:45.675 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:40:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:40:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:40:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:40:45.676 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:45.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:45.677 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:40:45.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:45.677 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:40:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:45.678 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:40:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:45.679 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:45.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:45.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:45.680 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:40:45.680 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:45.681 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:45.682 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:45.682 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:45.682 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:45.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:40:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:45.683 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:45.684 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:45.684 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:45.684 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:45.684 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:45.685 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:40:45.686 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:45.686 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:45.686 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:45.686 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:45.686 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:45.687 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:45.688 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:45.688 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:40:45.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:45.688 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:40:45.689 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:40:45.689 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:40:45.689 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:45.689 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:40:45.690 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:40:50.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:50.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:50.692 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:40:50.742 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:40:50.742 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:40:50.799 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:40:50.799 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:40:50.799 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:40:50.800 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:40:50.800 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:40:50.800 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:40:50.800 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:50.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:50.801 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:40:50.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:50.801 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:40:50.802 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:50.802 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:50.802 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:50.802 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:50.803 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:40:50.803 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:50.803 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:50.803 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:50.804 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:50.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:50.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:50.804 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:40:50.804 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:50.805 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:50.805 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:50.805 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:50.805 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:50.806 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:50.806 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:50.806 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:50.806 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:50.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:50.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:50.807 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:40:50.807 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:50.807 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:50.808 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:50.808 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:50.808 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:50.808 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:50.808 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:50.809 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:50.809 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:50.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:50.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:50.809 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:40:50.810 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:50.810 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:50.810 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:50.810 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:50.811 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:50.811 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:50.811 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:50.811 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:50.811 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:50.812 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:50.812 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:40:50.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:50.812 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:40:50.813 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:40:50.813 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:40:50.813 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:50.813 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:40:50.814 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:40:55.816 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:55.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:40:55.817 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:40:55.864 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:40:55.865 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:40:55.915 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:40:55.915 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:40:55.916 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:40:55.916 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:40:55.916 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:40:55.916 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:40:55.917 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:55.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:55.917 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:40:55.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:55.917 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:40:55.918 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:55.918 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:55.918 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:55.919 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:55.919 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:40:55.919 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:55.920 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:55.920 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:55.920 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:55.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:55.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:55.922 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:40:55.922 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:55.923 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:55.923 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:55.923 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:55.924 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:55.924 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:55.924 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:55.924 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:55.925 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:55.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:55.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:55.925 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:40:55.926 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:55.926 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:55.926 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:55.927 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:55.927 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:55.927 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:55.927 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:55.928 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:55.928 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:55.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:55.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:40:55.929 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:40:55.929 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:40:55.929 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:40:55.929 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:40:55.930 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:40:55.930 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:40:55.930 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:40:55.930 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:40:55.931 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:40:55.931 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:55.931 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:55.931 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:40:55.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:40:55.932 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:40:55.932 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:40:55.932 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:40:55.932 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:40:55.933 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:40:55.934 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:41:00.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:00.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:00.936 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:00.983 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:00.984 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:01.034 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:41:01.035 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:41:01.035 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:41:01.035 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:41:01.035 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:41:01.036 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:01.036 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:01.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:01.036 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:41:01.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:01.037 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:41:01.037 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:01.037 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:01.038 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:01.038 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:01.038 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:41:01.038 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:01.039 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:01.039 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:01.039 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:01.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:01.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:01.040 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:41:01.040 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:01.040 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:01.040 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:01.041 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:01.041 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:01.041 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:01.041 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:01.042 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:01.042 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:01.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:01.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:01.042 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:41:01.043 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:01.043 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:01.043 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:01.043 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:01.044 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:01.044 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:01.044 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:01.044 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:01.044 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:01.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:01.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:01.045 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:41:01.045 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:01.046 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:01.046 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:01.046 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:01.046 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:01.047 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:01.047 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:01.047 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:01.047 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:01.047 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:01.048 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:41:01.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:01.048 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:41:01.048 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:41:01.049 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:41:01.049 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:01.049 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:41:01.050 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:41:06.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:06.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:06.052 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:06.099 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:06.100 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:06.150 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:41:06.150 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:41:06.151 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:41:06.151 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:41:06.151 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:41:06.151 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:06.152 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:06.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:06.152 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:41:06.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:06.153 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:41:06.153 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:06.153 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:06.153 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:06.153 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:06.154 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:41:06.154 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:06.154 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:06.154 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:06.155 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:06.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:06.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:06.155 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:41:06.155 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:06.156 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:06.156 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:06.156 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:06.156 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:06.156 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:06.157 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:06.157 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:06.157 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:06.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:06.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:06.158 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:41:06.158 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:06.158 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:06.158 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:06.159 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:06.159 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:06.159 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:06.159 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:06.160 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:06.160 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:06.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:06.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:06.160 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:41:06.161 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:06.161 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:06.161 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:06.161 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:06.161 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:06.162 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:06.162 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:06.162 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:06.162 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:06.163 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:06.163 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:41:06.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:06.163 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:41:06.163 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:41:06.164 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:41:06.164 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:06.164 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:41:06.165 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:41:11.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:11.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:11.167 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:11.217 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:11.217 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:11.273 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:41:11.274 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:41:11.274 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:41:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:41:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:41:11.275 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:11.276 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:11.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:11.276 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:41:11.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:11.277 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:41:11.277 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:11.277 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:11.277 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:11.277 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:11.278 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:41:11.278 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:11.278 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:11.278 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:41:11.279 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:11.280 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:11.280 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:11.280 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:11.280 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:11.280 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:11.281 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:11.281 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:11.281 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:11.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:41:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:11.282 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:11.283 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:11.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:11.285 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:41:11.285 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:11.285 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:11.285 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:11.286 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:11.286 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:11.286 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:11.286 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:11.286 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:11.287 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:11.287 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:11.287 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:41:11.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:11.287 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:41:11.288 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:41:11.288 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:41:11.288 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:11.288 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:41:11.289 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:41:16.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:16.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:16.292 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:16.339 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:16.340 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:16.390 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:41:16.390 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:41:16.390 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:41:16.391 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:41:16.391 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:41:16.391 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:16.391 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:16.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:16.392 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:41:16.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:16.392 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:41:16.393 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:16.393 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:16.393 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:16.393 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:16.393 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:41:16.394 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:16.394 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:16.394 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:16.394 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:16.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:16.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:16.395 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:41:16.395 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:16.395 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:16.396 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:16.396 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:16.396 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:16.396 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:16.397 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:16.397 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:16.397 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:16.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:16.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:16.398 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:41:16.398 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:16.398 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:16.398 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:16.399 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:16.399 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:16.399 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:16.399 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:16.400 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:16.400 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:16.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:16.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:16.400 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:41:16.401 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:16.401 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:16.401 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:16.401 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:16.401 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:16.402 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:16.402 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:16.402 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:16.402 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:16.403 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:16.403 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:41:16.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:16.403 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:41:16.403 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:41:16.404 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:41:16.404 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:16.404 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:41:16.405 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:41:21.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:21.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:21.408 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:21.456 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:21.457 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:21.507 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:41:21.508 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:41:21.508 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:41:21.508 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:41:21.509 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:41:21.509 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:21.509 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:21.509 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:21.510 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:41:21.510 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:21.510 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:41:21.510 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:21.510 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:21.511 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:21.511 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:21.511 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:41:21.511 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:21.512 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:21.512 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:21.512 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:21.512 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:21.512 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:21.513 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:41:21.513 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:21.513 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:21.514 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:21.514 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:21.514 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:21.514 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:21.514 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:21.515 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:21.515 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:21.515 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:21.515 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:21.516 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:41:21.516 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:21.516 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:21.516 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:21.516 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:21.517 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:21.517 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:21.517 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:21.517 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:21.517 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:21.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:21.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:21.518 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:41:21.518 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:21.519 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:21.519 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:21.519 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:21.519 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:21.519 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:21.520 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:21.521 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:21.521 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:21.521 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:21.521 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:41:21.521 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:21.522 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:41:21.522 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:41:21.522 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:41:21.522 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:21.523 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:41:21.524 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:41:26.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:26.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:26.526 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:26.573 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:26.574 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:26.624 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:41:26.624 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:41:26.625 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:41:26.625 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:41:26.626 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:41:26.626 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:26.627 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:26.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:26.627 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:41:26.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:26.628 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:41:26.628 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:26.628 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:26.629 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:26.629 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:26.630 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:41:26.630 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:26.630 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:26.630 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:26.631 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:26.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:26.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:26.631 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:41:26.632 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:26.632 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:26.632 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:26.632 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:26.632 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:26.633 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:26.633 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:26.633 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:26.633 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:26.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:26.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:26.634 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:41:26.634 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:26.634 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:26.635 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:26.635 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:26.635 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:26.635 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:26.635 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:26.636 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:26.636 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:26.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:26.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:26.637 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:41:26.637 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:26.637 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:26.637 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:26.638 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:26.638 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:26.638 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:26.638 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:26.639 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:26.639 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:26.639 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:26.639 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:41:26.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:26.640 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:41:26.640 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:41:26.640 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:41:26.641 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:26.641 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:41:26.642 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:41:31.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:31.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:31.644 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:31.692 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:31.693 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:31.743 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:41:31.743 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:41:31.744 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:41:31.744 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:41:31.744 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:41:31.744 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:31.745 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:31.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:31.745 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:41:31.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:31.746 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:41:31.746 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:31.746 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:31.746 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:31.747 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:31.747 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:41:31.747 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:31.747 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:31.748 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:31.748 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:31.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:31.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:31.749 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:41:31.749 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:31.749 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:31.749 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:31.749 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:31.750 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:31.750 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:31.750 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:31.750 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:31.751 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:31.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:31.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:31.751 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:41:31.752 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:31.752 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:31.752 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:31.752 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:31.753 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:31.753 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:31.753 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:31.753 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:31.753 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:31.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:31.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:31.754 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:41:31.755 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:31.755 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:31.755 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:31.756 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:31.756 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:31.756 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:31.757 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:31.757 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:31.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:31.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:31.758 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:41:31.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:31.758 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:41:31.758 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:41:31.759 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:41:31.759 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:31.759 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:41:31.760 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:41:36.761 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:36.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:36.762 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:36.810 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:36.811 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:36.861 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:41:36.862 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:41:36.862 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:41:36.862 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:41:36.863 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:41:36.863 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:36.863 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:36.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:36.863 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:41:36.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:36.864 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:41:36.864 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:36.864 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:36.865 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:36.865 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:36.865 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:41:36.865 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:36.866 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:36.866 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:36.866 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:36.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:36.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:36.867 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:41:36.867 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:36.867 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:36.867 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:36.868 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:36.868 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:36.868 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:36.868 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:36.868 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:36.869 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:36.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:36.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:36.869 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:41:36.870 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:36.870 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:36.870 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:36.870 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:36.870 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:36.871 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:36.871 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:36.871 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:36.871 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:36.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:36.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:36.872 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:41:36.872 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:36.872 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:36.873 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:36.873 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:36.873 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:36.873 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:36.873 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:36.874 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:36.874 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:36.874 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:36.874 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:41:36.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:36.875 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:41:36.875 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:41:36.875 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:41:36.875 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:36.876 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:41:36.877 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1631821184.045455_63.1 target= 2021-09-16T19:41:41.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:41.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:41.879 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:41.929 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:41.929 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:41.980 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:41:41.980 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:41:41.980 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:41:41.981 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:41:41.981 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:41:41.981 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:41.981 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:41.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:41.982 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:41:41.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:41.983 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:41:41.983 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:41.983 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:41.983 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:41:41.984 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:41.984 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:41:41.984 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:41.984 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:41:41.984 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:41.985 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:41.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:41.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:41.986 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:41:41.987 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:41.987 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:41.987 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:41.987 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:41.988 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:41.988 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:41.988 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:41.988 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:41.989 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:41.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:41.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:41.989 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:41:41.989 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:41.990 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:41.990 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:41.990 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:41.991 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:41.991 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:41.991 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:41.991 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:41.992 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:41.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:41.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:41.992 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:41:41.992 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:41.993 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:41.993 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:41.993 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:41.993 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:41.993 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:41.994 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:41.994 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:41.994 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:41.994 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:41.995 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:41:41.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:41.995 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:41:41.995 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:41:41.995 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:41:41.996 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:41.996 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:41:41.997 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:41:41.998 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:41:41.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:41.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:42.000 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:42.051 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:42.052 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:42.112 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:41:42.113 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:41:42.113 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:41:42.113 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:41:42.113 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:41:42.114 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:42.114 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:41:42.115 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:42.115 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:41:42.115 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:42.115 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:41:42.116 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:41:42.116 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:41:42.116 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:41:42.116 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:41:42.117 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:41:42.117 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:41:42.117 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:41:42.117 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:41:42.117 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:42.118 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:42.118 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:42.118 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:41:42.118 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:41:42.119 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:41:42.119 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:41:42.119 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:41:42.120 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:41:42.120 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:41:42.120 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:41:42.120 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:41:42.121 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:42.121 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:42.121 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:42.121 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:41:42.121 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:41:42.122 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:41:42.122 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:41:42.122 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:41:42.122 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:41:42.123 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:41:42.123 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:41:42.123 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:41:42.123 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:42.124 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:42.124 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:42.125 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:41:42.125 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:41:42.125 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:41:42.125 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:41:42.126 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:41:42.126 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:41:42.126 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:41:42.126 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:41:42.127 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:41:42.127 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:42.128 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:42.128 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:41:42.128 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:42.128 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:41:42.129 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:41:42.129 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:41:42.129 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:42.129 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:41:42.131 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-16T19:41:42.131 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-16T19:41:42.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:42.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:42.132 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:42.176 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:42.176 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:42.221 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:41:42.221 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:41:42.222 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:41:42.222 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:41:42.222 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:41:42.222 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:42.222 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:42.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:42.223 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:41:42.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:42.223 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:41:42.224 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:42.224 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:42.224 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:41:42.224 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:42.224 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:41:42.225 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:42.225 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:41:42.225 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:42.225 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:42.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:42.226 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:41:42.226 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:42.227 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:42.227 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:42.227 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:42.227 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:42.227 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:42.228 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:42.228 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:42.228 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:42.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:42.229 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:41:42.229 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:42.229 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:42.229 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:42.230 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:42.230 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:42.230 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:42.231 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:42.231 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:42.231 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:42.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:42.232 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:41:42.232 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:42.232 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:42.232 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:42.233 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:42.233 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:42.233 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:42.233 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:42.233 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:42.234 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.234 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.234 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:41:42.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:42.235 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:41:42.235 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:41:42.235 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:41:42.235 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.235 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:41:42.237 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-16T19:41:42.237 INFO:rgw_multi.tests:meta checkpoint 2021-09-16T19:41:42.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:42.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:42.238 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:42.282 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:42.282 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:42.337 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:41:42.350 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:41:42.351 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:42.351 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:41:42.351 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:41:42.351 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:42.352 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:42.352 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:41:42.352 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:41:42.352 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:42.352 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:42.353 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:41:42.353 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:41:42.353 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:42.353 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:42.353 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:41:42.354 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:41:42.354 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:42.354 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:41:42.355 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:41:42.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:42.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:42.356 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:42.398 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:42.398 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:42.443 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:41:42.444 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:41:42.444 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:41:42.444 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:41:42.445 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:41:42.445 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:42.445 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:42.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:42.446 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:41:42.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:42.446 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:41:42.446 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:42.446 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:42.447 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:41:42.447 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:42.447 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:41:42.447 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:42.448 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:41:42.448 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:42.448 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:42.448 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:42.449 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:41:42.449 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:42.449 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:42.449 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:42.450 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:42.450 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:42.450 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:42.450 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:42.451 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:42.451 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:42.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:42.452 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:41:42.452 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:42.452 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:42.452 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:42.452 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:42.453 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:42.453 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:42.453 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:42.453 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:42.453 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:42.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:42.454 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:41:42.454 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:42.455 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:42.455 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:42.455 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:42.456 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:42.456 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:42.456 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:42.456 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:42.457 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.457 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.457 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:41:42.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:42.457 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:41:42.458 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:41:42.458 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:41:42.458 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.458 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:41:42.459 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:41:42.460 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:41:42.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:42.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:42.461 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:42.504 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:42.505 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:42.564 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:41:42.565 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:41:42.565 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:41:42.565 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:41:42.566 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:41:42.566 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:42.566 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:41:42.566 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:42.567 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:41:42.567 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:42.567 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:41:42.567 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:41:42.567 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:41:42.568 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:41:42.568 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:41:42.568 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:41:42.568 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:41:42.568 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:41:42.569 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:41:42.569 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:42.569 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:42.569 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:42.570 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:41:42.570 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:41:42.570 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:41:42.570 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:41:42.571 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:41:42.571 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:41:42.571 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:41:42.571 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:41:42.572 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:41:42.572 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:42.572 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:42.572 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:42.572 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:41:42.573 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:41:42.573 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:41:42.573 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:41:42.573 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:41:42.573 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:41:42.574 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:41:42.574 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:41:42.574 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:41:42.574 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:42.574 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:42.575 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:42.575 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:41:42.575 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:41:42.575 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:41:42.576 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:41:42.576 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:41:42.576 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:41:42.576 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:41:42.576 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:41:42.577 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:41:42.577 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:42.577 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:42.577 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:41:42.577 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:42.578 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:41:42.578 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:41:42.578 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:41:42.578 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:42.578 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:41:42.580 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-16T19:41:42.580 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-16T19:41:42.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:42.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:42.581 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:42.623 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:42.624 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:42.669 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:41:42.669 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:41:42.670 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:41:42.670 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:41:42.670 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:41:42.670 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:42.671 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:42.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:42.671 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:41:42.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:42.672 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:41:42.672 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:42.672 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:42.672 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:41:42.672 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:42.673 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:41:42.673 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:42.673 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:41:42.673 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:42.674 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:42.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:42.674 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:41:42.675 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:42.675 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:42.675 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:42.675 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:42.675 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:42.676 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:42.676 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:42.676 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:42.676 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:42.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:42.677 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:41:42.677 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:42.678 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:42.678 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:42.678 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:42.678 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:42.678 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:42.679 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:42.679 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:42.679 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:42.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:42.680 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:41:42.680 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:42.680 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:42.680 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:42.681 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:42.681 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:42.681 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:42.681 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:42.682 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:42.682 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.682 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.683 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:41:42.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:42.683 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:41:42.683 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:41:42.684 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:41:42.684 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:42.684 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:41:42.685 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-16T19:41:42.686 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph-c_4e2e64ee99b6cbb34351964aad0f9df4874dea6d/qa/../src/test/rgw/rgw_multi' with args=['tests.py', 'tests_ps.py'] 2021-09-16T19:41:42.725 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-1 2021-09-16T19:41:45.935 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=htrfwo-2 2021-09-16T19:41:49.614 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=htrfwo-3 2021-09-16T19:41:49.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:49.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:49.654 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:49.704 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:49.704 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:49.752 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:41:49.764 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:41:49.764 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:49.764 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:41:49.765 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:41:49.765 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:49.765 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:49.765 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:41:49.766 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:41:49.766 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:49.766 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:49.766 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:41:49.767 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:41:49.767 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:49.767 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:49.767 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821309.640838_75.1", 2021-09-16T19:41:49.768 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:41:49.640838Z" 2021-09-16T19:41:49.768 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:49.768 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:41:49.769 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:41:49.770 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:49.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:49.771 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:49.817 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:49.818 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:49.863 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:41:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:41:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:41:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:41:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:41:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:41:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:41:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:41:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:41:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:41:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:41:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:49.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:49.872 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:41:49.872 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:49.872 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:49.872 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:41:49.875 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:49.875 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:49.875 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821305.920291_64.1", 2021-09-16T19:41:49.875 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:49.875 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:49.876 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:49.876 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:41:45.920291Z", 2021-09-16T19:41:49.876 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:49.876 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:49.877 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:49.877 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:41:49.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:49.877 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:41:49.877 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:41:49.878 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:41:49.878 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:49.878 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:41:49.880 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1631821309.640838_75.1 target=1_1631821305.920291_64.1 2021-09-16T19:41:54.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:54.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:54.882 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:54.931 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:54.931 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:54.981 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:41:54.982 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:41:54.982 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:41:54.982 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:41:54.982 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:41:54.983 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:54.983 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:54.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:54.983 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:41:54.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:54.984 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:41:54.984 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:54.984 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:54.984 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:41:54.985 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:54.985 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:41:54.985 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:54.985 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:41:54.985 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:54.986 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:54.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:54.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:54.986 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:41:54.987 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:54.987 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:54.987 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:54.987 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:54.988 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:54.988 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:54.988 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:54.988 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:54.988 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:54.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:54.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:54.989 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:41:54.989 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:54.990 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:54.990 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:54.990 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:54.990 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:54.990 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:54.991 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:54.991 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:54.991 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:54.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:54.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:54.992 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:41:54.992 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:54.992 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:54.992 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821309.640838_75.1", 2021-09-16T19:41:54.993 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:54.993 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:54.993 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:54.993 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:41:49.640838Z", 2021-09-16T19:41:54.994 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:54.994 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:54.994 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:54.994 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:41:54.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:54.995 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:41:54.995 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:41:54.995 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:41:54.995 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:54.996 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:41:54.997 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:41:54.997 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:41:54.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:54.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:54.999 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:55.049 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:55.049 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:55.106 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:41:55.106 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:41:55.107 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:41:55.107 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:41:55.108 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:41:55.109 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:55.109 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:41:55.109 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:55.110 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:41:55.110 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:55.110 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:41:55.110 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:41:55.111 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:41:55.111 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:41:55.111 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:41:55.111 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:41:55.111 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:41:55.112 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:41:55.112 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:41:55.112 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:55.112 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:55.113 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:55.113 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:41:55.113 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:41:55.113 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:41:55.114 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:41:55.114 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:41:55.114 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:41:55.114 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:41:55.114 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:41:55.115 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:41:55.115 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:55.115 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:55.115 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:55.116 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:41:55.116 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:41:55.116 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:41:55.116 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:41:55.116 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:41:55.117 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:41:55.117 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:41:55.117 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:41:55.117 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:41:55.118 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:55.118 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:55.118 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:55.118 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:41:55.118 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:41:55.119 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:41:55.119 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821309.640838_75.1", 2021-09-16T19:41:55.119 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:41:55.119 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:41:55.120 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:41:55.120 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:41:49.640838Z", 2021-09-16T19:41:55.120 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:41:55.120 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:55.120 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:55.121 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:41:55.121 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:55.121 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:41:55.121 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:41:55.122 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:41:55.122 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:55.122 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:41:55.123 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-16T19:41:55.124 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-16T19:41:55.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:55.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:55.125 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:55.168 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:55.169 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:55.213 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:41:55.214 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:41:55.214 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:41:55.214 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:41:55.214 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:41:55.215 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:55.215 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:55.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:55.215 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:41:55.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:55.216 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:41:55.216 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:55.216 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:55.216 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:41:55.217 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:55.217 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:41:55.217 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:55.217 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:41:55.217 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:55.218 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:55.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:55.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:55.218 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:41:55.219 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:55.219 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:55.219 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:55.219 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:55.219 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:55.220 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:55.220 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:55.220 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:55.220 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:55.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:55.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:55.221 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:41:55.221 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:55.221 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:55.221 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:55.222 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:55.222 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:55.222 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:55.222 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:55.223 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:55.223 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:55.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:55.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:55.223 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:41:55.224 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:55.224 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:55.224 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821309.640838_75.1", 2021-09-16T19:41:55.224 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:55.224 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:55.225 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:55.225 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:41:49.640838Z", 2021-09-16T19:41:55.225 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:55.225 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:55.225 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:55.226 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:41:55.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:55.226 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:41:55.226 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:41:55.226 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:41:55.227 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:55.227 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:41:55.228 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-16T19:41:55.247 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2021-09-16T19:41:55.248 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-4 2021-09-16T19:41:55.268 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=htrfwo-5 2021-09-16T19:41:55.290 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=htrfwo-6 2021-09-16T19:41:55.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:55.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:55.323 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:55.365 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:55.365 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:55.415 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:41:55.426 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:41:55.426 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:55.426 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:41:55.427 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:41:55.427 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:55.427 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:55.427 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:41:55.428 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:41:55.428 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:55.428 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:55.428 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:41:55.429 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:41:55.429 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:41:55.429 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:41:55.429 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821315.306827_87.1", 2021-09-16T19:41:55.429 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:41:55.306827Z" 2021-09-16T19:41:55.430 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:41:55.430 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:41:55.431 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:41:55.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:55.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:41:55.433 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:41:55.476 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:41:55.476 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:41:55.523 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:41:55.523 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:41:55.523 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:41:55.524 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:41:55.524 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:41:55.524 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:41:55.524 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:55.524 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:55.525 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:41:55.525 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:55.525 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:41:55.525 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:55.526 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:55.526 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:41:55.526 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:55.526 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:41:55.527 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:55.527 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:41:55.527 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:55.527 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:55.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:55.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:55.528 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:41:55.528 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:55.529 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:55.529 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:55.529 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:55.529 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:55.529 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:55.530 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:55.530 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:55.530 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:55.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:55.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:55.531 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:41:55.531 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:55.531 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:55.532 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:41:55.532 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:55.532 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:55.532 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:55.532 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:41:55.533 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:55.533 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:55.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:55.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:41:55.534 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:41:55.534 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:41:55.534 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:41:55.534 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821315.257797_76.1", 2021-09-16T19:41:55.534 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:41:55.535 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:41:55.535 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:41:55.535 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:41:55.257797Z", 2021-09-16T19:41:55.535 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:41:55.536 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:55.536 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:55.536 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:41:55.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:41:55.537 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:41:55.537 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:41:55.537 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:41:55.537 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:41:55.537 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:41:55.539 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1631821315.306827_87.1 target=1_1631821315.257797_76.1 2021-09-16T19:42:00.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:00.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:00.541 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:00.590 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:00.591 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:00.649 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:00.650 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:00.650 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:00.651 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:00.651 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:00.651 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:00.652 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:00.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:00.653 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:00.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:00.653 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:00.654 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:00.654 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:00.655 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:00.655 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:00.655 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:00.656 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:00.656 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:00.656 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:00.657 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:00.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:00.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:00.658 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:00.658 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:00.659 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:00.659 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:00.659 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:00.660 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:00.660 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:00.661 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:00.661 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:00.661 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:00.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:00.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:00.663 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:00.663 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:00.663 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:00.664 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:00.664 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:00.664 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:00.665 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:00.665 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:00.666 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:00.666 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:00.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:00.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:00.667 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:00.668 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:00.668 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:00.669 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821315.306827_87.1", 2021-09-16T19:42:00.669 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:00.669 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:00.670 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:00.670 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:41:55.306827Z", 2021-09-16T19:42:00.671 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:00.671 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:00.672 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:00.672 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:00.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:00.673 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:00.673 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:00.674 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:00.674 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:00.674 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:00.677 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:42:00.678 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:42:00.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:00.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:00.680 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:00.730 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:00.731 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:00.786 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:42:00.786 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:42:00.787 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:42:00.787 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:42:00.787 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:42:00.787 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:00.788 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:00.788 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:00.788 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:42:00.788 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:00.789 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:42:00.789 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:00.789 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:00.789 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:00.790 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:00.790 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:42:00.790 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:00.790 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:00.791 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:00.791 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:00.791 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:00.791 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:00.791 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:42:00.792 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:00.792 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:00.792 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:00.792 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:00.793 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:00.793 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:00.793 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:42:00.793 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:00.794 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:00.794 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:00.794 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:00.794 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:42:00.795 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:00.795 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:00.795 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:00.795 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:00.795 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:00.796 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:00.796 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:42:00.796 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:00.796 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:00.797 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:00.797 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:00.797 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:42:00.797 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:00.797 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:00.798 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821315.306827_87.1", 2021-09-16T19:42:00.798 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:00.798 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:00.798 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:00.799 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:41:55.306827Z", 2021-09-16T19:42:00.799 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:00.799 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:00.799 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:00.799 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:42:00.800 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:00.800 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:42:00.800 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:42:00.800 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:42:00.800 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:00.801 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:42:00.802 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-16T19:42:00.802 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-16T19:42:00.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:00.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:00.804 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:00.850 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:00.851 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:00.899 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:00.900 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:00.900 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:00.900 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:00.901 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:00.901 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:00.901 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:00.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:00.901 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:00.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:00.902 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:00.902 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:00.902 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:00.902 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:00.903 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:00.903 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:00.903 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:00.903 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:00.904 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:00.904 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:00.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:00.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:00.904 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:00.905 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:00.905 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:00.905 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:00.905 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:00.906 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:00.906 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:00.906 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:00.906 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:00.906 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:00.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:00.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:00.907 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:00.907 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:00.907 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:00.908 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:00.908 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:00.908 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:00.908 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:00.908 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:00.909 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:00.909 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:00.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:00.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:00.910 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:00.910 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:00.910 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:00.910 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821315.306827_87.1", 2021-09-16T19:42:00.910 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:00.911 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:00.911 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:00.911 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:41:55.306827Z", 2021-09-16T19:42:00.911 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:00.911 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:00.912 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:00.912 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:00.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:00.912 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:00.912 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:00.913 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:00.913 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:00.913 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:00.914 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-16T19:42:01.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:01.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:01.156 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:01.197 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:01.197 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:01.243 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:42:01.257 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:42:01.258 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:01.258 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:01.259 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:42:01.259 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:01.260 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:01.261 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:01.262 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:42:01.262 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:01.263 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:01.263 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:01.263 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:42:01.264 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:01.264 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:01.264 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821315.306827_87.1", 2021-09-16T19:42:01.265 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:41:55.306827Z" 2021-09-16T19:42:01.265 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:01.266 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:42:01.267 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:42:01.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:01.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:01.269 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:01.314 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:01.314 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:01.362 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:01.366 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:01.366 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:01.366 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:01.366 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:01.367 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:01.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:01.373 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:01.373 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821315.306827_87.1", 2021-09-16T19:42:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:41:55.306827Z", 2021-09-16T19:42:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.376 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:01.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:01.376 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:01.376 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:01.377 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:01.377 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.377 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:01.378 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:42:01.379 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:42:01.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:01.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:01.380 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:01.422 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:01.423 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:01.490 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:42:01.490 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:42:01.490 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:42:01.490 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:42:01.491 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:42:01.491 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:01.491 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:01.491 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:01.491 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:42:01.492 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:01.492 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:42:01.492 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:01.492 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:01.493 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:01.493 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:01.493 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:42:01.493 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:01.493 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:01.494 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:01.494 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:01.494 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:01.494 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:01.494 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:42:01.495 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:01.495 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:01.495 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:01.496 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:01.496 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:01.496 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:01.496 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:42:01.497 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:01.497 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:01.497 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:01.497 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:01.497 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:42:01.498 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:01.498 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:01.498 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:01.498 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:01.499 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:01.499 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:01.499 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:42:01.499 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:01.499 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:01.500 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:01.500 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:01.500 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:42:01.500 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:01.501 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:01.501 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821315.306827_87.1", 2021-09-16T19:42:01.501 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:01.501 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:01.501 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:01.502 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:41:55.306827Z", 2021-09-16T19:42:01.502 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:01.502 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:01.502 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:01.503 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:42:01.503 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:01.503 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:42:01.503 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:42:01.503 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:42:01.504 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:01.504 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:42:01.505 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-16T19:42:01.505 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-16T19:42:01.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:01.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:01.508 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:01.551 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:01.551 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:01.596 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:01.597 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:01.597 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:01.597 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:01.597 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:01.598 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:01.598 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:01.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:01.598 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:01.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:01.599 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:01.599 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:01.599 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:01.599 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:01.600 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:01.600 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:01.600 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:01.600 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:01.600 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:01.601 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:01.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:01.601 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:01.601 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:01.602 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:01.602 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:01.602 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:01.602 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:01.603 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:01.603 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:01.603 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:01.603 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:01.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:01.604 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:01.604 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:01.604 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:01.604 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:01.604 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:01.605 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:01.605 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:01.605 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:01.605 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:01.605 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:01.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:01.606 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:01.606 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:01.606 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:01.607 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821315.306827_87.1", 2021-09-16T19:42:01.607 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:01.607 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:01.607 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:01.607 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:41:55.306827Z", 2021-09-16T19:42:01.608 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:01.608 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.608 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.608 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:01.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:01.609 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:01.609 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:01.609 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:01.609 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.609 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:01.611 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-16T19:42:01.629 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2021-09-16T19:42:01.630 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-7 2021-09-16T19:42:01.645 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=htrfwo-8 2021-09-16T19:42:01.670 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=htrfwo-9 2021-09-16T19:42:01.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:01.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:01.697 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:01.740 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:01.741 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:01.791 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:42:01.808 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:42:01.808 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:01.809 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:01.809 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:42:01.809 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:01.809 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:01.810 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:01.810 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:42:01.810 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:01.810 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:01.810 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:01.811 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:42:01.811 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:01.811 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:01.811 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821321.684049_99.1", 2021-09-16T19:42:01.812 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:01.684049Z" 2021-09-16T19:42:01.812 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:01.812 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:42:01.813 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:42:01.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:01.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:01.815 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:01.858 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:01.859 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:01.904 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:01.904 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:01.904 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:01.906 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:01.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:01.906 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:01.906 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:01.908 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:01.908 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:01.908 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:01.911 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:01.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:01.911 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821321.637179_88.1", 2021-09-16T19:42:01.915 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:01.915 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:01.915 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:01.915 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:01.637179Z", 2021-09-16T19:42:01.916 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:01.916 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.916 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.916 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:01.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:01.917 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:01.917 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:01.917 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:01.917 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:01.918 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:01.919 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1631821321.684049_99.1 target=1_1631821321.637179_88.1 2021-09-16T19:42:06.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:06.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:06.922 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:06.972 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:06.972 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:07.026 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:07.026 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:07.026 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:07.027 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:07.027 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:07.027 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:07.027 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:07.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:07.028 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:07.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:07.028 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:07.029 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:07.029 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:07.029 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:07.029 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:07.029 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:07.030 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:07.030 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:07.030 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:07.031 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:07.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:07.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:07.031 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:07.032 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:07.032 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:07.032 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:07.032 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:07.033 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:07.033 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:07.033 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:07.033 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:07.034 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:07.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:07.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:07.034 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:07.035 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:07.035 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:07.035 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:07.035 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:07.035 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:07.036 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:07.036 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:07.036 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:07.037 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:07.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:07.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:07.037 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:07.038 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:07.038 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:07.038 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821321.680014_97.1", 2021-09-16T19:42:07.038 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:07.039 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:07.039 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:07.039 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:01.680014Z", 2021-09-16T19:42:07.039 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:07.039 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:07.040 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:07.040 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:07.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:07.040 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:07.041 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:07.041 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:07.041 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:07.041 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:07.043 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1631821321.684049_99.1 target=1_1631821321.680014_97.1 2021-09-16T19:42:12.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:12.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:12.046 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:12.093 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:12.094 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:12.144 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:12.144 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:12.145 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:12.145 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:12.145 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:12.145 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:12.146 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:12.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:12.146 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:12.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:12.146 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:12.147 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:12.147 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:12.147 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:12.147 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:12.148 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:12.148 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:12.148 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:12.148 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:12.148 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:12.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:12.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:12.149 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:12.149 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:12.150 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:12.150 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:12.150 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:12.150 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:12.150 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:12.151 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:12.151 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:12.151 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:12.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:12.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:12.152 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:12.152 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:12.152 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:12.152 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:12.153 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:12.153 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:12.153 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:12.153 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:12.153 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:12.154 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:12.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:12.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:12.154 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:12.155 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:12.155 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:12.155 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821321.684049_99.1", 2021-09-16T19:42:12.155 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:12.155 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:12.156 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:12.156 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:01.684049Z", 2021-09-16T19:42:12.156 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:12.156 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:12.157 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:12.157 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:12.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:12.157 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:12.158 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:12.158 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:12.158 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:12.158 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:12.159 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:42:12.160 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:42:12.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:12.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:12.161 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:12.210 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:12.210 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:12.265 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:42:12.265 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:42:12.265 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:42:12.266 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:42:12.266 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:42:12.266 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:12.266 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:12.267 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:12.267 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:42:12.267 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:12.267 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:42:12.267 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:12.268 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:12.268 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:12.268 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:12.268 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:42:12.269 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:12.269 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:12.269 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:12.269 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:12.269 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:12.270 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:12.270 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:42:12.270 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:12.270 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:12.271 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:12.271 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:12.271 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:12.271 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:12.272 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:42:12.272 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:12.272 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:12.272 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:12.273 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:12.273 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:42:12.273 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:12.273 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:12.273 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:12.274 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:12.274 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:12.274 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:12.274 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:42:12.275 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:12.275 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:12.275 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:12.275 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:12.275 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:42:12.276 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:12.276 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:12.276 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821321.680014_97.1", 2021-09-16T19:42:12.276 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:12.277 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:12.277 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:12.277 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:01.680014Z", 2021-09-16T19:42:12.277 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:12.277 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:12.278 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:12.278 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:42:12.278 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:12.278 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:42:12.279 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:42:12.279 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:42:12.279 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:12.279 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:42:12.280 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1631821321.684049_99.1 target=1_1631821321.680014_97.1 2021-09-16T19:42:17.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:17.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:17.284 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:17.332 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:17.332 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:17.391 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:42:17.391 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:42:17.391 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:42:17.392 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:42:17.392 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:42:17.392 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:17.392 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:17.392 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:17.393 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:42:17.393 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:17.393 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:42:17.393 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:17.394 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:17.394 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:17.394 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:17.394 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:42:17.395 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:17.395 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:17.395 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:17.395 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:17.395 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:17.396 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:17.396 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:42:17.396 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:17.396 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:17.396 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:17.397 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:17.397 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:17.397 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:17.397 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:42:17.398 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:17.398 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:17.398 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:17.398 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:17.398 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:42:17.399 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:17.399 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:17.399 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:17.399 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:17.400 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:17.400 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:17.400 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:42:17.400 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:17.400 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:17.401 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:17.401 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:17.401 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:42:17.401 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:17.402 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:17.402 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821321.684049_99.1", 2021-09-16T19:42:17.402 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:17.402 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:17.403 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:17.403 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:01.684049Z", 2021-09-16T19:42:17.403 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:17.403 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:17.403 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:17.404 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:42:17.404 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:17.404 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:42:17.404 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:42:17.404 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:42:17.405 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:17.405 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:42:17.406 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-16T19:42:17.406 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-16T19:42:17.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:17.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:17.409 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:17.456 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:17.457 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:17.506 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:17.507 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:17.507 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:17.508 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:17.509 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:17.510 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:17.510 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:17.511 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:17.512 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:17.512 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:17.512 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:17.512 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:17.513 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:17.514 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:17.514 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:17.514 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:17.515 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:17.516 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:17.517 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:17.518 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821321.684049_99.1", 2021-09-16T19:42:17.519 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:17.520 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:17.520 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:17.520 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:01.684049Z", 2021-09-16T19:42:17.520 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:17.520 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:17.521 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:17.521 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:17.521 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:17.522 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:17.522 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:17.522 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:17.522 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:17.522 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:17.524 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-16T19:42:17.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:17.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:17.670 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:17.711 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:17.711 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:17.746 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:42:17.757 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:42:17.758 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:17.758 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:17.758 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:42:17.758 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:17.759 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:17.759 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:17.759 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:42:17.759 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:17.760 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:17.760 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:17.760 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:42:17.760 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:17.760 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:17.761 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:42:17.761 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:17.633408Z" 2021-09-16T19:42:17.761 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:17.761 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:42:17.762 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:42:17.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:17.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:17.764 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:17.807 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:17.807 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:17.852 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:17.852 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:17.853 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:17.853 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:17.853 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:17.853 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:17.854 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:17.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:17.854 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:17.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:17.855 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:17.855 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:17.855 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:17.855 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:17.856 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:17.856 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:17.856 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:17.856 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:17.856 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:17.857 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:17.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:17.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:17.857 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:17.858 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:17.858 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:17.858 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:17.858 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:17.859 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:17.859 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:17.859 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:17.859 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:17.859 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:17.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:17.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:17.860 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:17.860 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:17.860 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:17.861 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:17.861 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:17.861 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:17.861 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:17.862 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:17.862 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:17.862 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:17.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:17.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:17.863 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:17.863 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:17.864 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:17.864 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.560153_100.1", 2021-09-16T19:42:17.864 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:17.864 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:17.865 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:17.865 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.560153Z", 2021-09-16T19:42:17.865 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:17.865 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:17.865 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:17.866 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:17.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:17.866 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:17.866 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:17.866 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:17.867 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:17.867 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:17.868 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1631821337.633408_105.1 target=1_1631821337.560153_100.1 2021-09-16T19:42:22.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:22.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:22.871 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:22.919 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:22.920 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:22.971 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:22.971 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:22.971 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:22.972 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:22.972 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:22.972 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:22.972 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:22.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:22.973 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:22.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:22.973 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:22.973 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:22.974 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:22.974 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:22.974 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:22.974 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:22.974 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:22.975 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:22.975 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:22.975 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:22.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:22.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:22.976 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:22.976 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:22.976 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:22.976 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:22.977 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:22.977 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:22.977 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:22.977 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:22.977 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:22.978 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:22.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:22.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:22.978 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:22.978 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:22.979 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:22.979 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:22.979 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:22.979 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:22.979 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:22.980 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:22.980 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:22.980 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:22.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:22.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:22.981 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:22.981 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:22.981 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:22.981 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:42:22.981 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:22.982 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:22.982 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:22.982 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:42:22.982 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:22.982 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:22.983 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:22.983 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:22.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:22.983 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:22.983 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:22.984 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:22.984 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:22.984 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:22.985 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:42:22.986 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:42:22.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:22.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:22.987 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:23.038 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:23.038 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:23.090 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:42:23.090 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:42:23.091 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:42:23.091 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:42:23.091 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:42:23.091 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:23.092 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:23.092 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:23.092 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:42:23.092 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:23.092 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:42:23.093 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:23.093 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:23.093 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:23.093 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:23.094 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:42:23.094 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:23.094 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:23.094 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:23.094 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:23.095 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:23.095 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:23.095 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:42:23.095 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:23.096 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:23.096 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:23.096 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:23.096 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:23.096 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:23.097 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:42:23.097 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:23.097 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:23.097 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:23.098 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:23.098 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:42:23.098 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:23.098 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:23.099 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:23.099 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:23.099 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:23.099 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:23.099 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:42:23.100 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:23.100 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:23.100 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:23.100 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:23.100 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:42:23.101 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:23.101 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:23.101 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:42:23.101 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:23.102 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:23.102 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:23.102 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:42:23.102 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:23.102 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:23.103 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:23.103 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:42:23.103 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:23.103 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:42:23.103 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:42:23.104 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:42:23.104 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:23.104 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:42:23.105 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-16T19:42:23.106 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-16T19:42:23.106 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:23.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:23.107 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:23.150 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:23.151 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:23.197 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:23.197 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:23.197 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:23.198 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:23.198 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:23.198 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:23.198 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:23.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:23.199 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:23.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:23.199 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:23.200 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:23.200 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:23.200 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:23.200 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:23.201 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:23.201 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:23.201 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:23.201 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:23.201 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:23.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:23.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:23.202 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:23.202 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:23.203 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:23.203 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:23.203 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:23.203 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:23.203 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:23.204 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:23.204 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:23.204 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:23.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:23.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:23.205 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:23.205 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:23.205 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:23.205 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:23.206 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:23.206 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:23.206 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:23.206 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:23.206 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:23.207 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:23.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:23.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:23.207 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:23.208 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:23.208 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:23.208 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:42:23.208 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:23.209 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:23.209 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:23.209 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:42:23.209 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:23.209 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:23.210 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:23.210 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:23.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:23.210 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:23.211 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:23.211 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:23.211 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:23.212 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:23.213 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-16T19:42:23.223 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2021-09-16T19:42:23.224 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-10 2021-09-16T19:42:23.237 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=htrfwo-11 2021-09-16T19:42:23.264 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=htrfwo-12 2021-09-16T19:42:23.322 INFO:tasks.rgw.c1.client.1.smithi109.stdout:2021-09-16T19:42:23.321+0000 7f2511691700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:23.323 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2021-09-16T19:42:23.322+0000 7f0e0a1ab700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:23.324 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T19:42:23.322+0000 7f2fd3368700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:23.324 INFO:tasks.rgw.c1.client.1.smithi109.stdout:2021-09-16T19:42:23.322+0000 7f251e6ab700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:23.522 INFO:tasks.rgw.c1.client.1.smithi109.stdout:2021-09-16T19:42:23.520+0000 7f251cea8700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:23.522 INFO:tasks.rgw.c1.client.1.smithi109.stdout:2021-09-16T19:42:23.520+0000 7f251feae700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:23.523 INFO:tasks.rgw.c1.client.1.smithi109.stdout:2021-09-16T19:42:23.520+0000 7f251c6a7700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:23.524 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T19:42:23.521+0000 7f2fd7b71700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:23.524 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T19:42:23.521+0000 7f2fdeb7f700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:23.524 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T19:42:23.521+0000 7f2fd9b75700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:23.525 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2021-09-16T19:42:23.521+0000 7f0e079a6700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:23.525 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2021-09-16T19:42:23.521+0000 7f0e081a7700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:23.525 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2021-09-16T19:42:23.521+0000 7f0e069a4700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:23.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:23.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:23.852 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:23.909 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:23.910 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:23.992 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:42:24.034 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:42:24.034 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:24.035 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:24.035 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:42:24.035 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:24.035 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:24.036 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:24.036 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:42:24.036 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:24.036 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:24.036 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821343.280103_81.1", 2021-09-16T19:42:24.037 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:23.280103Z" 2021-09-16T19:42:24.037 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:24.037 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:24.037 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:42:24.038 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:17.633408Z" 2021-09-16T19:42:24.038 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:24.038 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:42:24.039 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:42:24.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:24.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:24.041 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:24.084 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:24.085 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:24.131 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:24.131 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:24.132 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:24.132 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:24.133 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:24.133 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:24.133 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:24.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:24.134 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:24.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:24.135 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:24.136 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:24.136 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:24.137 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:24.137 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:24.138 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:24.138 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:24.138 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:24.139 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:24.139 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:24.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:24.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:24.141 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:24.141 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:24.142 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:24.142 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:24.143 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:24.143 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:24.143 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:24.144 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:24.144 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:24.145 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:24.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:24.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:24.146 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:24.146 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:24.147 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:24.147 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821343.230007_70.1", 2021-09-16T19:42:24.148 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:24.148 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:24.149 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:24.149 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:23.230007Z", 2021-09-16T19:42:24.149 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:24.150 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:24.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:24.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:24.151 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:24.152 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:24.152 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:24.153 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:42:24.153 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:24.153 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:24.154 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:24.154 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:42:24.155 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:24.155 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:24.156 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:24.156 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:24.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:24.157 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:24.157 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:24.158 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:24.158 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:24.159 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:24.161 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821343.280103_81.1 target=1_1631821343.230007_70.1 2021-09-16T19:42:24.722 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T19:42:24.721+0000 7f2fd8372700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:24.923 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T19:42:24.922+0000 7f2fd536c700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:24.923 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T19:42:24.922+0000 7f2fe0382700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:24.924 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T19:42:24.922+0000 7f2fda376700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:29.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:29.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:29.163 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:29.211 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:29.212 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:29.263 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:29.264 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:29.264 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:29.265 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:29.265 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:29.265 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:29.266 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:29.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:29.267 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:29.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:29.268 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:29.268 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:29.269 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:29.269 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:29.269 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:29.270 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:29.270 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:29.271 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:29.271 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:29.272 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:29.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:29.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:29.273 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:29.273 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:29.274 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:29.274 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:29.274 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:29.275 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:29.275 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:29.275 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:29.276 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:29.276 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:29.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:29.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:29.277 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:29.278 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:29.278 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:29.279 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821343.280103_81.1", 2021-09-16T19:42:29.279 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:29.279 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:29.280 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:29.280 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:23.280103Z", 2021-09-16T19:42:29.281 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:29.281 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:29.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:29.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:29.282 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:29.283 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:29.283 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:29.284 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:42:29.284 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:29.284 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:29.285 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:29.285 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:42:29.286 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:29.286 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:29.286 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:29.286 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:29.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:29.287 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:29.287 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:29.288 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:29.288 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:29.288 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:29.290 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:42:29.291 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:42:29.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:29.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:29.294 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:29.339 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:29.340 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:29.409 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:42:29.409 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:42:29.410 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:42:29.410 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:42:29.410 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:42:29.410 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:29.411 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:29.411 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:29.411 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:42:29.411 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:29.411 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:42:29.412 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:29.412 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:29.412 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:29.412 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:29.413 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:42:29.413 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:29.413 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:29.413 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:29.413 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:29.414 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:29.414 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:29.414 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:42:29.414 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:29.415 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:29.415 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:29.415 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:29.415 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:29.415 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:29.416 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:42:29.416 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:29.416 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:29.416 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:29.417 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:29.417 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:42:29.417 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:29.417 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:29.417 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821343.280103_81.1", 2021-09-16T19:42:29.418 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:29.418 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:29.418 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:29.418 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:23.280103Z", 2021-09-16T19:42:29.418 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:29.419 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:29.419 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:29.419 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:29.419 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:42:29.419 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:29.420 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:29.420 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:42:29.420 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:29.420 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:29.421 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:29.421 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:42:29.421 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:29.421 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:29.421 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:29.421 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:42:29.422 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:29.422 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:42:29.422 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:42:29.422 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:42:29.423 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:29.423 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:42:29.424 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-16T19:42:29.425 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-16T19:42:29.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:29.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:29.426 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:29.470 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:29.470 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:29.517 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:29.517 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:29.518 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:29.518 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:29.518 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:29.518 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:29.519 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:29.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:29.519 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:29.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:29.519 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:29.520 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:29.520 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:29.520 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:29.520 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:29.521 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:29.521 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:29.521 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:29.521 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:29.521 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:29.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:29.522 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:29.522 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:29.522 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:29.523 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:29.523 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:29.523 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:29.523 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:29.523 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:29.524 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:29.524 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:29.524 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:29.524 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:29.524 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:29.525 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:29.525 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:29.525 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:29.525 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821343.280103_81.1", 2021-09-16T19:42:29.526 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:29.526 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:29.526 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:29.526 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:23.280103Z", 2021-09-16T19:42:29.526 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:29.527 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:29.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:29.527 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:29.527 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:29.528 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:29.528 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:29.528 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:42:29.528 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:29.529 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:29.529 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:29.529 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:42:29.529 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:29.529 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:29.530 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:29.530 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:29.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:29.530 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:29.530 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:29.531 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:29.531 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:29.531 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:29.532 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-16T19:42:29.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:29.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:29.534 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:29.622 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:29.622 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:29.639 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:29.639+0000 7f2049588380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:29.639 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:29.639+0000 7f2049588380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:29.640 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:29.639+0000 7f2049588380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:42:29.640 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:29.639+0000 7f2049588380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:29.664 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:29.663+0000 7f2049588380 1 bucket sync caught up with source: 2021-09-16T19:42:29.664 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000001.8.5, , 00000000001.12.5, , , , 00000000001.9.5, 00000000001.14.5, , ] 2021-09-16T19:42:29.664 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000001.8.5, , 00000000001.12.5, , , , 00000000001.9.5, 00000000001.14.5, , ] 2021-09-16T19:42:29.665 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:29.663+0000 7f2049588380 0 bucket checkpoint complete 2021-09-16T19:42:29.676 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-10 zones={test-zone2, test-zone1} 2021-09-16T19:42:29.681 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:29.686 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-16T19:42:29.687 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-16T19:42:29.687 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-16T19:42:29.687 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:29.688 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:29.691 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-16T19:42:29.691 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-16T19:42:29.692 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-16T19:42:29.692 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:29.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:29.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:29.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:29.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:29.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:30.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:30.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:30.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:30.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:30.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:30.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:30.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:30.673 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-10 zones={test-zone2, test-zone1} 2021-09-16T19:42:30.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:30.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:30.675 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:30.726 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:30.727 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:30.744 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:30.743+0000 7fb47c7ba380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:30.745 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:30.743+0000 7fb47c7ba380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:30.745 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:30.743+0000 7fb47c7ba380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:42:30.745 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:30.743+0000 7fb47c7ba380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:30.775 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:30.774+0000 7fb47c7ba380 1 bucket sync caught up with source: 2021-09-16T19:42:30.775 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000001.8.5, , 00000000001.12.5, , , , 00000000001.9.5, 00000000001.14.5, , ] 2021-09-16T19:42:30.776 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000001.8.5, , 00000000001.12.5, , , , 00000000001.9.5, 00000000001.14.5, , ] 2021-09-16T19:42:30.776 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:30.774+0000 7fb47c7ba380 0 bucket checkpoint complete 2021-09-16T19:42:30.793 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-10 zones={test-zone3, test-zone1} 2021-09-16T19:42:30.799 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:30.802 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-16T19:42:30.802 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-16T19:42:30.803 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-16T19:42:30.803 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:30.804 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:30.807 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-16T19:42:30.807 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-16T19:42:30.808 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-16T19:42:30.808 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:30.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:30.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:30.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:31.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:31.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:31.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:31.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:31.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:31.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:31.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:31.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:31.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:31.886 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-10 zones={test-zone3, test-zone1} 2021-09-16T19:42:31.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:31.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:31.888 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:31.943 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:31.944 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:31.963 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:31.963+0000 7fa221f38380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:31.964 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:31.963+0000 7fa221f38380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:31.964 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:31.963+0000 7fa221f38380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:42:31.964 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:31.963+0000 7fa221f38380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:31.989 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:31.988+0000 7fa221f38380 1 bucket sync caught up with source: 2021-09-16T19:42:31.989 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000001.8.5, , 00000000001.12.5, , , , 00000000001.9.5, 00000000001.14.5, , ] 2021-09-16T19:42:31.989 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000001.8.5, , 00000000001.12.5, , , , 00000000001.9.5, 00000000001.14.5, , ] 2021-09-16T19:42:31.990 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:31.988+0000 7fa221f38380 0 bucket checkpoint complete 2021-09-16T19:42:32.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:32.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:32.003 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:32.054 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:32.054 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:32.073 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:32.071+0000 7fb2186ea380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:32.073 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:32.071+0000 7fb2186ea380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:32.073 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:32.071+0000 7fb2186ea380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:42:32.074 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:32.071+0000 7fb2186ea380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:32.110 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:32.108+0000 7fb2186ea380 1 bucket sync caught up with source: 2021-09-16T19:42:32.110 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000001.84.5, , 00000000001.75.5, , , , 00000000001.42.5, 00000000001.94.5, , ] 2021-09-16T19:42:32.111 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000001.84.5, , 00000000001.75.5, , , , 00000000001.42.5, 00000000001.94.5, , ] 2021-09-16T19:42:32.111 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:32.108+0000 7fb2186ea380 0 bucket checkpoint complete 2021-09-16T19:42:32.131 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-11 zones={test-zone1, test-zone2} 2021-09-16T19:42:32.136 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:32.139 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-16T19:42:32.140 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-16T19:42:32.140 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-16T19:42:32.141 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:32.141 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:32.145 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-16T19:42:32.146 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-16T19:42:32.147 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-16T19:42:32.148 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:32.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:32.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:32.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:32.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:32.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:32.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:32.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:32.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:32.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:32.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:32.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:33.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:33.153 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-11 zones={test-zone1, test-zone2} 2021-09-16T19:42:33.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:33.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:33.155 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:33.209 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:33.209 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:33.229 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:33.227+0000 7f63e0bab380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:33.229 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:33.227+0000 7f63e0bab380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:33.229 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:33.227+0000 7f63e0bab380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:42:33.229 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:33.227+0000 7f63e0bab380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:33.270 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:33.268+0000 7f63e0bab380 1 bucket sync caught up with source: 2021-09-16T19:42:33.270 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000001.84.5, , 00000000001.75.5, , , , 00000000001.42.5, 00000000001.94.5, , ] 2021-09-16T19:42:33.270 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000001.84.5, , 00000000001.75.5, , , , 00000000001.42.5, 00000000001.94.5, , ] 2021-09-16T19:42:33.271 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:33.268+0000 7f63e0bab380 0 bucket checkpoint complete 2021-09-16T19:42:33.282 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-11 zones={test-zone3, test-zone2} 2021-09-16T19:42:33.287 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:33.290 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-16T19:42:33.291 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-16T19:42:33.291 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-16T19:42:33.292 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:33.292 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:33.339 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-16T19:42:33.339 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-16T19:42:33.340 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-16T19:42:33.340 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:33.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:33.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:33.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:33.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:33.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:33.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:33.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:33.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:34.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:34.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:34.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:34.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:34.390 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-11 zones={test-zone3, test-zone2} 2021-09-16T19:42:34.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:34.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:34.392 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:34.447 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:34.448 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:34.467 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:34.466+0000 7f9e6b753380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:34.467 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:34.466+0000 7f9e6b753380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:34.468 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:34.466+0000 7f9e6b753380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:42:34.468 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:34.466+0000 7f9e6b753380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:34.508 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:34.506+0000 7f9e6b753380 1 bucket sync caught up with source: 2021-09-16T19:42:34.508 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000001.22.5, , 00000000001.18.5, , , , 00000000001.52.5, 00000000001.100.5, , ] 2021-09-16T19:42:34.508 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000001.22.5, , 00000000001.18.5, , , , 00000000001.52.5, 00000000001.100.5, , ] 2021-09-16T19:42:34.509 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:34.506+0000 7f9e6b753380 0 bucket checkpoint complete 2021-09-16T19:42:34.526 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-12 zones={test-zone1, test-zone3} 2021-09-16T19:42:34.531 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:34.535 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-16T19:42:34.535 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-16T19:42:34.536 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-16T19:42:34.536 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:34.537 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:34.540 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-16T19:42:34.540 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-16T19:42:34.541 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-16T19:42:34.541 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:34.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:34.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:34.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:34.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:34.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:34.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:35.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:35.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:35.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:35.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:35.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:35.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:35.599 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-12 zones={test-zone1, test-zone3} 2021-09-16T19:42:35.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:35.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:35.601 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:35.655 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:35.656 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:35.676 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:35.674+0000 7f58f928a380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:35.676 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:35.674+0000 7f58f928a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:35.676 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:35.674+0000 7f58f928a380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:42:35.676 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:35.674+0000 7f58f928a380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:35.701 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:35.699+0000 7f58f928a380 1 bucket sync caught up with source: 2021-09-16T19:42:35.701 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000001.22.5, , 00000000001.18.5, , , , 00000000001.52.5, 00000000001.100.5, , ] 2021-09-16T19:42:35.701 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000001.22.5, , 00000000001.18.5, , , , 00000000001.52.5, 00000000001.100.5, , ] 2021-09-16T19:42:35.701 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:35.699+0000 7f58f928a380 0 bucket checkpoint complete 2021-09-16T19:42:35.713 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-12 zones={test-zone2, test-zone3} 2021-09-16T19:42:35.718 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:35.721 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-16T19:42:35.722 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-16T19:42:35.722 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-16T19:42:35.723 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:35.723 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:35.726 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-16T19:42:35.726 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-16T19:42:35.727 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-16T19:42:35.727 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:35.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:35.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:35.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-16T19:42:35.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:36.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:36.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-16T19:42:36.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:36.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:36.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-16T19:42:36.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:36.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:36.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:36.786 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-12 zones={test-zone2, test-zone3} 2021-09-16T19:42:36.787 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2021-09-16T19:42:36.787 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-13 2021-09-16T19:42:36.806 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=htrfwo-14 2021-09-16T19:42:36.835 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=htrfwo-15 2021-09-16T19:42:36.928 INFO:tasks.rgw.c1.client.1.smithi109.stdout:2021-09-16T19:42:36.927+0000 7f251deaa700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:36.930 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2021-09-16T19:42:36.928+0000 7f0e0219b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:36.930 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T19:42:36.928+0000 7f2fd3368700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:37.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:37.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:37.017 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:37.048 INFO:tasks.rgw.c1.client.1.smithi109.stdout:2021-09-16T19:42:37.046+0000 7f251cea8700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:37.063 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:37.063 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:37.065 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2021-09-16T19:42:37.064+0000 7f0e091a9700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:37.109 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T19:42:37.107+0000 7f2fd8372700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:42:37.112 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:42:37.131 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:42:37.131 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:37.131 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:37.132 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:42:37.132 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:37.132 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:37.132 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:37.133 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:42:37.133 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:37.133 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:37.133 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821356.855814_105.1", 2021-09-16T19:42:37.134 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:36.855814Z" 2021-09-16T19:42:37.134 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:37.134 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:37.134 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:42:37.134 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:17.633408Z" 2021-09-16T19:42:37.135 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:37.135 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:42:37.136 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:42:37.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:37.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:37.138 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:37.187 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:37.187 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:37.239 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:37.239 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:37.240 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:37.240 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:37.240 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:37.240 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:37.241 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:37.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:37.242 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:37.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:37.242 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:37.243 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:37.243 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:37.243 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:37.243 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:37.243 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:37.244 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:37.244 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:37.244 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:37.244 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:37.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:37.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:37.245 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:37.245 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:37.245 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:37.246 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:37.246 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:37.246 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:37.246 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:37.247 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:37.247 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:37.247 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:37.247 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:37.247 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:37.248 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:37.248 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:37.248 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:37.248 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821356.796429_94.1", 2021-09-16T19:42:37.249 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:37.249 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:37.249 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:37.249 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:36.796429Z", 2021-09-16T19:42:37.249 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:37.250 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:37.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:37.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:37.250 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:37.251 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:37.251 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:37.251 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:42:37.251 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:37.252 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:37.252 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:37.252 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:42:37.252 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:37.252 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:37.253 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:37.253 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:37.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:37.253 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:37.254 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:37.254 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:37.254 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:37.254 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:37.255 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821356.855814_105.1 target=1_1631821356.796429_94.1 2021-09-16T19:42:42.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:42.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:42.259 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:42.306 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:42.306 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:42.357 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821356.855814_105.1", 2021-09-16T19:42:42.366 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:42.366 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:42.366 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:42.366 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:36.855814Z", 2021-09-16T19:42:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:42:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:42:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:42.371 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:42.371 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:42.371 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:42.372 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:42:42.373 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:42:42.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:42.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:42.374 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:42.420 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:42.421 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:42.483 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:42:42.484 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:42:42.484 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:42:42.484 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:42:42.484 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:42:42.485 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:42.485 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:42.485 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:42.485 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:42:42.486 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:42.486 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:42:42.486 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:42.486 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:42.487 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:42.487 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:42.487 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:42:42.487 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:42.488 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:42.488 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:42.488 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:42.488 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:42.489 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:42.489 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:42:42.489 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:42.489 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:42.489 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:42.490 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:42.490 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:42.490 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:42.490 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:42:42.491 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:42.491 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:42.491 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:42.491 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:42.492 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:42:42.492 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:42.492 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:42.492 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821356.820117_99.1", 2021-09-16T19:42:42.492 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:42.493 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:42.493 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:42.493 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:36.820117Z", 2021-09-16T19:42:42.494 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:42.494 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:42.494 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:42.494 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:42.494 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:42:42.495 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:42.495 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:42.495 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:42:42.495 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:42.496 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:42.496 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:42.496 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:42:42.496 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:42.496 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:42.497 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:42.497 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:42:42.497 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:42.497 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:42:42.497 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:42:42.498 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:42:42.498 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:42.498 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:42:42.500 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821356.855814_105.1 target=1_1631821356.820117_99.1 2021-09-16T19:42:47.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:47.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:47.503 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:47.553 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:47.553 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:47.615 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:42:47.615 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:42:47.615 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:42:47.616 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:42:47.616 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:42:47.616 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:47.617 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:47.617 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:47.617 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:42:47.617 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:47.617 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:42:47.618 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:47.618 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:47.618 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:47.618 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:47.619 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:42:47.619 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:47.619 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:47.619 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:47.619 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:47.620 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:47.620 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:47.620 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:42:47.620 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:47.620 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:47.621 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:47.621 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:47.621 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:47.621 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:47.622 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:42:47.622 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:47.622 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:47.622 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:47.622 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:47.623 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:42:47.623 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:47.623 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:47.623 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821356.855814_105.1", 2021-09-16T19:42:47.624 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:47.624 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:47.624 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:47.624 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:36.855814Z", 2021-09-16T19:42:47.624 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:47.625 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:47.625 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:47.625 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:47.625 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:42:47.625 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:42:47.626 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:42:47.626 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:42:47.626 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:42:47.626 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:42:47.626 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:42:47.627 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:42:47.627 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:42:47.627 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:47.627 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:47.628 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:42:47.628 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:47.628 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:42:47.628 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:42:47.629 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:42:47.629 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:47.629 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:42:47.630 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-16T19:42:47.631 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-16T19:42:47.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:47.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:47.632 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:47.679 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:47.680 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:47.729 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:47.730 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:47.730 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:47.730 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:47.730 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:47.731 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:47.731 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:47.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:47.731 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:47.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:47.732 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:47.732 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:47.732 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:47.732 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:47.732 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:47.733 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:47.733 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:47.733 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:47.733 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:47.734 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:47.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:47.734 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:47.734 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:47.734 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:47.735 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:47.735 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:47.735 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:47.735 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:47.736 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:47.736 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:47.736 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:47.736 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:47.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:47.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:47.737 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:47.737 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:47.737 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:47.737 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821356.855814_105.1", 2021-09-16T19:42:47.738 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:47.738 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:47.738 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:47.738 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:36.855814Z", 2021-09-16T19:42:47.739 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:47.739 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:47.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:47.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:47.739 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:47.740 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:47.740 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:47.740 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:42:47.740 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:47.741 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:47.741 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:47.741 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:42:47.741 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:47.741 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:47.742 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:47.742 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:47.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:47.742 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:47.743 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:47.743 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:47.743 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:47.743 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:47.745 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-16T19:42:47.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:47.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:47.746 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:47.796 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:47.797 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:47.814 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:47.812+0000 7f9925ea2380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:47.814 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:47.812+0000 7f9925ea2380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:47.814 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:47.812+0000 7f9925ea2380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:42:47.814 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:47.812+0000 7f9925ea2380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:47.837 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:47.836+0000 7f9925ea2380 1 bucket sync caught up with source: 2021-09-16T19:42:47.838 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000001.4.5, , , , , , , , , ] 2021-09-16T19:42:47.838 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000001.4.5, , , , , , , , , ] 2021-09-16T19:42:47.838 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:47.836+0000 7f9925ea2380 0 bucket checkpoint complete 2021-09-16T19:42:47.850 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-13 zones={test-zone2, test-zone1} 2021-09-16T19:42:47.855 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:47.899 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:47.900 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:47.903 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:47.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:48.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:48.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:48.198 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-13 zones={test-zone2, test-zone1} 2021-09-16T19:42:48.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:48.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:48.200 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:48.250 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:48.251 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:48.267 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:48.266+0000 7f95a1521380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:48.268 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:48.266+0000 7f95a1521380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:48.268 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:48.266+0000 7f95a1521380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:42:48.268 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:48.266+0000 7f95a1521380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:48.305 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:48.303+0000 7f95a1521380 1 bucket sync caught up with source: 2021-09-16T19:42:48.305 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000001.4.5, , , , , , , , , ] 2021-09-16T19:42:48.305 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000001.4.5, , , , , , , , , ] 2021-09-16T19:42:48.305 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:48.303+0000 7f95a1521380 0 bucket checkpoint complete 2021-09-16T19:42:48.322 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-13 zones={test-zone3, test-zone1} 2021-09-16T19:42:48.327 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:48.371 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:48.371 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:48.374 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:48.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:48.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:48.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:48.665 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-13 zones={test-zone3, test-zone1} 2021-09-16T19:42:48.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:48.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:48.667 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:48.716 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:48.717 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:48.734 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:48.732+0000 7f5658320380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:48.734 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:48.732+0000 7f5658320380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:48.735 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:48.732+0000 7f5658320380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:42:48.735 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:48.732+0000 7f5658320380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:48.757 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:48.756+0000 7f5658320380 1 bucket sync caught up with source: 2021-09-16T19:42:48.758 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000001.4.5, , , , , , , , , ] 2021-09-16T19:42:48.758 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000001.4.5, , , , , , , , , ] 2021-09-16T19:42:48.758 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:48.756+0000 7f5658320380 0 bucket checkpoint complete 2021-09-16T19:42:48.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:48.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:48.771 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:48.821 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:48.821 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:48.838 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:48.837+0000 7f83fa340380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:48.839 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:48.837+0000 7f83fa340380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:48.839 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:48.837+0000 7f83fa340380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:42:48.839 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:48.837+0000 7f83fa340380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:48.870 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:48.868+0000 7f83fa340380 1 bucket sync caught up with source: 2021-09-16T19:42:48.870 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000001.117.5, , , , , , , , , ] 2021-09-16T19:42:48.870 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000001.117.5, , , , , , , , , ] 2021-09-16T19:42:48.871 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:48.868+0000 7f83fa340380 0 bucket checkpoint complete 2021-09-16T19:42:48.882 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-14 zones={test-zone1, test-zone2} 2021-09-16T19:42:48.887 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:48.931 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:48.931 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:48.934 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:48.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:49.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:49.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:49.225 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-14 zones={test-zone1, test-zone2} 2021-09-16T19:42:49.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:49.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:49.227 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:49.277 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:49.277 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:49.295 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:49.293+0000 7f708b7b1380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:49.295 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:49.293+0000 7f708b7b1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:49.296 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:49.293+0000 7f708b7b1380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:42:49.296 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:49.293+0000 7f708b7b1380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:49.329 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:49.328+0000 7f708b7b1380 1 bucket sync caught up with source: 2021-09-16T19:42:49.329 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000001.117.5, , , , , , , , , ] 2021-09-16T19:42:49.329 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000001.117.5, , , , , , , , , ] 2021-09-16T19:42:49.330 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:49.328+0000 7f708b7b1380 0 bucket checkpoint complete 2021-09-16T19:42:49.347 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-14 zones={test-zone3, test-zone2} 2021-09-16T19:42:49.352 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:49.395 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:49.395 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:49.398 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:49.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:49.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:49.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:49.690 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-14 zones={test-zone3, test-zone2} 2021-09-16T19:42:49.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:49.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:49.691 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:49.741 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:49.741 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:49.758 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:49.757+0000 7efc36d9d380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:49.758 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:49.757+0000 7efc36d9d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:49.759 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:49.757+0000 7efc36d9d380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:42:49.759 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:49.757+0000 7efc36d9d380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:49.785 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:49.784+0000 7efc36d9d380 1 bucket sync caught up with source: 2021-09-16T19:42:49.785 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000001.55.5, , , , , , , , , ] 2021-09-16T19:42:49.786 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000001.55.5, , , , , , , , , ] 2021-09-16T19:42:49.786 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:49.784+0000 7efc36d9d380 0 bucket checkpoint complete 2021-09-16T19:42:49.803 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-15 zones={test-zone1, test-zone3} 2021-09-16T19:42:49.808 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:49.851 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:49.851 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:49.854 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:49.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:49.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:50.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:50.181 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-15 zones={test-zone1, test-zone3} 2021-09-16T19:42:50.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:50.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:50.183 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:50.237 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:50.237 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:50.257 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:50.255+0000 7f07f5bd3380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:50.257 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:50.255+0000 7f07f5bd3380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:50.257 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:50.255+0000 7f07f5bd3380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:42:50.257 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:50.255+0000 7f07f5bd3380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:50.282 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:50.281+0000 7f07f5bd3380 1 bucket sync caught up with source: 2021-09-16T19:42:50.283 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000001.55.5, , , , , , , , , ] 2021-09-16T19:42:50.283 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000001.55.5, , , , , , , , , ] 2021-09-16T19:42:50.283 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:50.281+0000 7f07f5bd3380 0 bucket checkpoint complete 2021-09-16T19:42:50.295 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-15 zones={test-zone2, test-zone3} 2021-09-16T19:42:50.300 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:50.347 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:50.347 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:50.350 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-16T19:42:50.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:50.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:50.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-16T19:42:50.641 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-15 zones={test-zone2, test-zone3} 2021-09-16T19:42:50.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:50.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:50.651 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:50.701 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:50.701 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:50.718 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:50.716+0000 7f0c524ce380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:50.718 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:50.716+0000 7f0c524ce380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:50.719 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:50.716+0000 7f0c524ce380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:42:50.719 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:50.716+0000 7f0c524ce380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:50.743 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:50.741+0000 7f0c524ce380 1 bucket sync caught up with source: 2021-09-16T19:42:50.743 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000002.6.5, , , , , , , , , ] 2021-09-16T19:42:50.743 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000002.6.5, , , , , , , , , ] 2021-09-16T19:42:50.744 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:50.741+0000 7f0c524ce380 0 bucket checkpoint complete 2021-09-16T19:42:50.755 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-13 zones={test-zone2, test-zone1} 2021-09-16T19:42:50.760 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:50.803 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:50.853 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-13 zones={test-zone2, test-zone1} 2021-09-16T19:42:50.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:50.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:50.855 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:50.906 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:50.907 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:50.925 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:50.924+0000 7fe1b8192380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:50.925 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:50.924+0000 7fe1b8192380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:50.925 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:50.924+0000 7fe1b8192380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:42:50.925 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:50.924+0000 7fe1b8192380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:50.949 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:50.949+0000 7fe1b8192380 1 bucket sync caught up with source: 2021-09-16T19:42:50.949 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000002.6.5, , , , , , , , , ] 2021-09-16T19:42:50.950 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000002.6.5, , , , , , , , , ] 2021-09-16T19:42:50.950 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:50.949+0000 7fe1b8192380 0 bucket checkpoint complete 2021-09-16T19:42:50.961 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-13 zones={test-zone3, test-zone1} 2021-09-16T19:42:50.966 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:51.011 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:51.061 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-13 zones={test-zone3, test-zone1} 2021-09-16T19:42:51.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:51.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:51.063 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:51.114 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:51.197 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:51.200 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:51.129+0000 7f82e3264380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:51.200 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:51.129+0000 7f82e3264380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:51.200 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:51.129+0000 7f82e3264380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:42:51.201 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:51.129+0000 7f82e3264380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:51.201 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:51.152+0000 7f82e3264380 1 bucket sync caught up with source: 2021-09-16T19:42:51.201 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000002.6.5, , , , , , , , , ] 2021-09-16T19:42:51.202 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000002.6.5, , , , , , , , , ] 2021-09-16T19:42:51.202 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:51.152+0000 7f82e3264380 0 bucket checkpoint complete 2021-09-16T19:42:51.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:51.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:51.243 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:51.298 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:51.298 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:51.318 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:51.317+0000 7fe5927e7380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:51.319 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:51.317+0000 7fe5927e7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:51.319 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:51.317+0000 7fe5927e7380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:42:51.319 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:51.317+0000 7fe5927e7380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:51.357 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:51.357+0000 7fe5927e7380 1 bucket sync caught up with source: 2021-09-16T19:42:51.357 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000002.199.5, , , , , , , , , ] 2021-09-16T19:42:51.358 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000002.199.5, , , , , , , , , ] 2021-09-16T19:42:51.358 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:51.357+0000 7fe5927e7380 0 bucket checkpoint complete 2021-09-16T19:42:51.369 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-14 zones={test-zone1, test-zone2} 2021-09-16T19:42:51.383 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:51.439 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:51.515 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-14 zones={test-zone1, test-zone2} 2021-09-16T19:42:51.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:51.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:51.517 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:51.571 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:51.571 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:51.589 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:51.588+0000 7f96968dd380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:51.589 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:51.588+0000 7f96968dd380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:51.590 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:51.588+0000 7f96968dd380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:42:51.590 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:51.588+0000 7f96968dd380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:51.624 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:51.623+0000 7f96968dd380 1 bucket sync caught up with source: 2021-09-16T19:42:51.625 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000002.199.5, , , , , , , , , ] 2021-09-16T19:42:51.625 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000002.199.5, , , , , , , , , ] 2021-09-16T19:42:51.625 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:51.623+0000 7f96968dd380 0 bucket checkpoint complete 2021-09-16T19:42:51.645 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-14 zones={test-zone3, test-zone2} 2021-09-16T19:42:51.657 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:51.704 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:51.758 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-14 zones={test-zone3, test-zone2} 2021-09-16T19:42:51.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:51.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:51.768 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:51.817 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:51.818 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:51.835 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:51.834+0000 7f7e8647e380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:51.835 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:51.834+0000 7f7e8647e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:51.835 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:51.834+0000 7f7e8647e380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:42:51.835 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:51.834+0000 7f7e8647e380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:51.858 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:51.857+0000 7f7e8647e380 1 waiting for incremental sync to catch up: 2021-09-16T19:42:51.858 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000001.55.5, , , , , , , , , ] 2021-09-16T19:42:51.858 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000002.107.5, , , , , , , , , ] 2021-09-16T19:42:56.859 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:56.857+0000 7f7e8647e380 1 bucket sync caught up with source: 2021-09-16T19:42:56.860 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000002.107.5, , , , , , , , , ] 2021-09-16T19:42:56.860 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000002.107.5, , , , , , , , , ] 2021-09-16T19:42:56.860 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:42:56.857+0000 7f7e8647e380 0 bucket checkpoint complete 2021-09-16T19:42:56.879 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-15 zones={test-zone1, test-zone3} 2021-09-16T19:42:56.885 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:56.931 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:56.982 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-15 zones={test-zone1, test-zone3} 2021-09-16T19:42:56.983 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:56.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:42:56.984 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:42:57.039 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:57.039 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:57.060 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:57.058+0000 7f356fbf8380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:42:57.060 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:57.058+0000 7f356fbf8380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:42:57.060 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:57.058+0000 7f356fbf8380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:42:57.061 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:57.058+0000 7f356fbf8380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:42:57.087 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:57.086+0000 7f356fbf8380 1 bucket sync caught up with source: 2021-09-16T19:42:57.088 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000002.107.5, , , , , , , , , ] 2021-09-16T19:42:57.088 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000002.107.5, , , , , , , , , ] 2021-09-16T19:42:57.088 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:42:57.086+0000 7f356fbf8380 0 bucket checkpoint complete 2021-09-16T19:42:57.101 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-15 zones={test-zone2, test-zone3} 2021-09-16T19:42:57.106 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:42:57.155 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:42:57.205 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-15 zones={test-zone2, test-zone3} 2021-09-16T19:42:57.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2021-09-16T19:42:57.207 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-16 2021-09-16T19:42:57.230 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=htrfwo-17 2021-09-16T19:42:57.262 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=htrfwo-18 2021-09-16T19:42:57.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:57.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:57.308 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:57.355 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:57.356 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:57.394 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:42:57.411 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:42:57.411 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:57.411 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:57.412 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:42:57.412 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:57.412 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:57.412 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:42:57.412 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:42:57.413 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:57.413 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:57.413 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821377.304162_123.1", 2021-09-16T19:42:57.413 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:57.304162Z" 2021-09-16T19:42:57.413 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:42:57.414 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:42:57.414 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:42:57.414 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:17.633408Z" 2021-09-16T19:42:57.415 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:42:57.415 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:42:57.416 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:42:57.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:57.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:42:57.418 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:42:57.462 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:42:57.463 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:42:57.511 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:42:57.512 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:42:57.512 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:42:57.512 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:42:57.512 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:42:57.513 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:42:57.513 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:57.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:57.513 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:42:57.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:57.514 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:42:57.514 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:57.515 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:57.515 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:42:57.515 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:57.515 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:42:57.515 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:57.516 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:42:57.516 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:57.516 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:57.516 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:57.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:57.517 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:42:57.517 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:57.517 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:57.517 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:42:57.518 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:57.518 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:57.518 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:57.518 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:42:57.518 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:57.519 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:57.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:57.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:57.519 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:42:57.520 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:57.520 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:57.520 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821377.217052_106.1", 2021-09-16T19:42:57.520 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:57.520 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:57.521 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:57.521 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:57.217052Z", 2021-09-16T19:42:57.521 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:57.521 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:57.521 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:57.522 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:42:57.522 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:42:57.522 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:42:57.522 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:42:57.523 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:42:57.523 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:42:57.523 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:42:57.523 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:42:57.523 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:42:57.524 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:42:57.524 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:57.524 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:57.524 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:42:57.524 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:42:57.524 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:42:57.525 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:42:57.525 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:42:57.525 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:42:57.525 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:42:57.527 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821377.304162_123.1 target=1_1631821377.217052_106.1 2021-09-16T19:43:02.528 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:43:02.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:43:02.529 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:43:02.576 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:02.577 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:02.629 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:43:02.629 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:43:02.630 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:43:02.630 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:43:02.630 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:43:02.631 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:43:02.631 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:02.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:02.631 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:43:02.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:02.632 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:43:02.632 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:02.632 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:02.632 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:43:02.633 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:02.633 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:43:02.633 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:02.633 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:43:02.633 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:02.634 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:02.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:02.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:02.634 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:43:02.635 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:02.635 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:02.635 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:43:02.636 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:02.636 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:43:02.636 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:02.636 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:43:02.636 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:02.637 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:02.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:02.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:02.637 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:43:02.638 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:02.638 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:02.638 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821377.304162_123.1", 2021-09-16T19:43:02.638 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:02.638 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:43:02.639 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:02.639 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:57.304162Z", 2021-09-16T19:43:02.639 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:02.639 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:02.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:02.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:02.640 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:43:02.640 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:02.640 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:02.641 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:43:02.641 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:02.641 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:43:02.641 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:02.642 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:43:02.642 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:02.642 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:02.642 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:02.643 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:43:02.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:02.643 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:43:02.643 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:43:02.643 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:43:02.644 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:02.644 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:43:02.645 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:43:02.646 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:43:02.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:43:02.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:43:02.647 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:43:02.696 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:02.696 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:02.757 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:43:02.757 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:43:02.758 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:43:02.758 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:43:02.758 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:43:02.759 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:43:02.759 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:43:02.759 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:43:02.759 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:43:02.760 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:43:02.760 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:43:02.760 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:43:02.760 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:43:02.760 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:43:02.761 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:43:02.761 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:43:02.761 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:43:02.761 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:43:02.762 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:43:02.762 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:43:02.762 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:43:02.762 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:43:02.763 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:43:02.763 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:43:02.763 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:43:02.763 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:43:02.763 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:43:02.764 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:43:02.764 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:43:02.764 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:43:02.764 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:43:02.765 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:43:02.765 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:43:02.765 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:43:02.765 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:43:02.765 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:43:02.766 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:43:02.766 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821377.304162_123.1", 2021-09-16T19:43:02.766 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:43:02.766 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:43:02.767 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:43:02.767 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:57.304162Z", 2021-09-16T19:43:02.767 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:43:02.767 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:43:02.767 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:43:02.768 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:43:02.768 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:43:02.768 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:43:02.768 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:43:02.769 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:43:02.769 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:43:02.769 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:43:02.769 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:43:02.770 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:43:02.770 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:43:02.770 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:43:02.771 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:43:02.771 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:43:02.771 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:43:02.771 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:43:02.771 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:43:02.772 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:43:02.772 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:43:02.772 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:43:02.773 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-16T19:43:02.774 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-16T19:43:02.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:43:02.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:43:02.775 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:43:02.818 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:02.818 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:02.864 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:43:02.864 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:43:02.865 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:43:02.865 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:43:02.865 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:43:02.865 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:43:02.866 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:02.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:02.866 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:43:02.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:02.867 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:43:02.867 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:02.867 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:02.867 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:43:02.867 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:02.868 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:43:02.868 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:02.868 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:43:02.868 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:02.869 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:02.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:02.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:02.869 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:43:02.870 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:02.870 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:02.870 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:43:02.870 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:02.871 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:43:02.871 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:02.871 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:43:02.871 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:02.872 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:02.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:02.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:02.872 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:43:02.873 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:02.873 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:02.873 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821377.304162_123.1", 2021-09-16T19:43:02.873 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:02.873 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:43:02.874 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:02.874 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:57.304162Z", 2021-09-16T19:43:02.874 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:02.874 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:02.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:02.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:02.875 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:43:02.875 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:02.876 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:02.876 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:43:02.876 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:02.876 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:43:02.876 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:02.877 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:43:02.877 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:02.877 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:02.877 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:02.878 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:43:02.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:02.878 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:43:02.878 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:43:02.878 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:43:02.879 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:02.879 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:43:02.880 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-16T19:43:02.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:02.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:02.928 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:02.977 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:02.977 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:02.995 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:02.993+0000 7fcb03600380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:02.995 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:02.993+0000 7fcb03600380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:02.995 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:02.993+0000 7fcb03600380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:43:02.996 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:02.993+0000 7fcb03600380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:03.020 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:03.018+0000 7fcb03600380 1 waiting for incremental sync to catch up: 2021-09-16T19:43:03.020 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , , , ] 2021-09-16T19:43:03.020 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , 00000000002.4.10, , , , , ] 2021-09-16T19:43:08.021 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:08.020+0000 7fcb03600380 1 bucket sync caught up with source: 2021-09-16T19:43:08.021 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , 00000000006.12.9, , , , , ] 2021-09-16T19:43:08.021 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , 00000000002.4.10, , , , , ] 2021-09-16T19:43:08.022 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:08.020+0000 7fcb03600380 0 bucket checkpoint complete 2021-09-16T19:43:08.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:08.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:08.038 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:08.093 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:08.093 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:08.112 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:08.111+0000 7fc20a017380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:08.113 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:08.111+0000 7fc20a017380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:08.113 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:08.111+0000 7fc20a017380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:43:08.113 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:08.111+0000 7fc20a017380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:08.153 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:08.151+0000 7fc20a017380 1 bucket sync caught up with source: 2021-09-16T19:43:08.153 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , 00000000006.12.9, , , , , ] 2021-09-16T19:43:08.154 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , 00000000006.12.9, , , , , ] 2021-09-16T19:43:08.154 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:08.151+0000 7fc20a017380 0 bucket checkpoint complete 2021-09-16T19:43:08.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:08.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:08.170 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:08.227 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:08.227 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:08.251 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:08.250+0000 7fdc602f2380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:08.251 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:08.250+0000 7fdc602f2380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:08.251 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:08.250+0000 7fdc602f2380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:43:08.251 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:08.250+0000 7fdc602f2380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:08.278 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:08.278+0000 7fdc602f2380 1 bucket sync caught up with source: 2021-09-16T19:43:08.278 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , 00000000006.12.9, , , , , ] 2021-09-16T19:43:08.279 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , 00000000006.12.9, , , , , ] 2021-09-16T19:43:08.279 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:08.278+0000 7fdc602f2380 0 bucket checkpoint complete 2021-09-16T19:43:08.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:08.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:08.340 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:08.391 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:08.391 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:08.408 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:08.407+0000 7f5a8e908380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:08.408 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:08.407+0000 7f5a8e908380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:08.408 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:08.407+0000 7f5a8e908380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:43:08.409 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:08.407+0000 7f5a8e908380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:08.447 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:08.446+0000 7f5a8e908380 1 waiting for incremental sync to catch up: 2021-09-16T19:43:08.448 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , , , ] 2021-09-16T19:43:08.448 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , 00000000002.234.10, , , , , ] 2021-09-16T19:43:13.449 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:13.447+0000 7f5a8e908380 1 bucket sync caught up with source: 2021-09-16T19:43:13.449 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , 00000000006.242.9, , , , , ] 2021-09-16T19:43:13.450 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , 00000000002.234.10, , , , , ] 2021-09-16T19:43:13.450 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:13.447+0000 7f5a8e908380 0 bucket checkpoint complete 2021-09-16T19:43:13.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:13.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:13.470 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:13.522 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:13.523 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:13.542 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:13.540+0000 7fee17441380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:13.542 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:13.540+0000 7fee17441380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:13.542 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:13.540+0000 7fee17441380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:43:13.543 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:13.540+0000 7fee17441380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:13.584 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:13.583+0000 7fee17441380 1 bucket sync caught up with source: 2021-09-16T19:43:13.585 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , 00000000006.242.9, , , , , ] 2021-09-16T19:43:13.585 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , 00000000006.242.9, , , , , ] 2021-09-16T19:43:13.585 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:13.583+0000 7fee17441380 0 bucket checkpoint complete 2021-09-16T19:43:13.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:13.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:13.648 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:13.698 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:13.698 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:13.715 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:13.714+0000 7feec79c9380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:13.715 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:13.714+0000 7feec79c9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:13.716 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:13.714+0000 7feec79c9380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:43:13.716 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:13.714+0000 7feec79c9380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:13.741 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:13.740+0000 7feec79c9380 1 waiting for incremental sync to catch up: 2021-09-16T19:43:13.742 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , , , ] 2021-09-16T19:43:13.742 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , 00000000002.93.10, , , , , ] 2021-09-16T19:43:18.743 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:18.742+0000 7feec79c9380 1 bucket sync caught up with source: 2021-09-16T19:43:18.744 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , 00000000005.99.9, , , , , ] 2021-09-16T19:43:18.744 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , 00000000002.93.10, , , , , ] 2021-09-16T19:43:18.744 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:18.742+0000 7feec79c9380 0 bucket checkpoint complete 2021-09-16T19:43:18.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:18.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:18.760 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:18.815 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:18.816 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:18.836 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:18.834+0000 7f84c2a45380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:18.836 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:18.834+0000 7f84c2a45380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:18.836 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:18.834+0000 7f84c2a45380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:43:18.836 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:18.834+0000 7f84c2a45380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:18.862 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:18.861+0000 7f84c2a45380 1 bucket sync caught up with source: 2021-09-16T19:43:18.862 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , 00000000005.99.9, , , , , ] 2021-09-16T19:43:18.863 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , 00000000005.99.9, , , , , ] 2021-09-16T19:43:18.863 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:18.861+0000 7f84c2a45380 0 bucket checkpoint complete 2021-09-16T19:43:20.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:20.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:20.326 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:20.377 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:20.377 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:20.395 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:20.393+0000 7f564b535380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:20.395 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:20.393+0000 7f564b535380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:20.395 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:20.393+0000 7f564b535380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:43:20.395 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:20.393+0000 7f564b535380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:20.420 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:20.418+0000 7f564b535380 1 bucket sync caught up with source: 2021-09-16T19:43:20.420 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000010.27.5, , 00000000003.9.9, , 00000000009.18.9, , , , , ] 2021-09-16T19:43:20.420 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000010.27.5, , 00000000003.9.9, , 00000000009.18.9, , , , , ] 2021-09-16T19:43:20.420 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:20.418+0000 7f564b535380 0 bucket checkpoint complete 2021-09-16T19:43:20.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:20.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:20.434 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:20.488 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:20.489 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:20.511 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.509+0000 7f8d98309380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:20.511 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.509+0000 7f8d98309380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:20.511 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.509+0000 7f8d98309380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:43:20.511 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.509+0000 7f8d98309380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:20.553 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.551+0000 7f8d98309380 1 bucket sync caught up with source: 2021-09-16T19:43:20.553 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000010.27.5, , 00000000003.9.9, , 00000000009.18.9, , , , , ] 2021-09-16T19:43:20.554 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000010.27.5, , 00000000003.9.9, , 00000000009.18.9, , , , , ] 2021-09-16T19:43:20.554 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.551+0000 7f8d98309380 0 bucket checkpoint complete 2021-09-16T19:43:20.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:20.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:20.567 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:20.616 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:20.616 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:20.633 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:20.631+0000 7f80ff92b380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:20.633 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:20.631+0000 7f80ff92b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:20.634 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:20.631+0000 7f80ff92b380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:43:20.634 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:20.631+0000 7f80ff92b380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:20.658 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:20.656+0000 7f80ff92b380 1 bucket sync caught up with source: 2021-09-16T19:43:20.658 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000010.27.5, , 00000000003.9.9, , 00000000009.18.9, , , , , ] 2021-09-16T19:43:20.658 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000010.27.5, , 00000000003.9.9, , 00000000009.18.9, , , , , ] 2021-09-16T19:43:20.658 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:20.656+0000 7f80ff92b380 0 bucket checkpoint complete 2021-09-16T19:43:20.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:20.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:20.671 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:20.721 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:20.721 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:20.739 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.737+0000 7feb0c8a9380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:20.739 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.737+0000 7feb0c8a9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:20.739 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.737+0000 7feb0c8a9380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:43:20.739 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.737+0000 7feb0c8a9380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:20.778 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.777+0000 7feb0c8a9380 1 bucket sync caught up with source: 2021-09-16T19:43:20.778 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000006.247.5, , 00000000010.129.5, , 00000000010.89.5, , , , , ] 2021-09-16T19:43:20.779 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000006.247.5, , 00000000010.129.5, , 00000000010.89.5, , , , , ] 2021-09-16T19:43:20.779 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.777+0000 7feb0c8a9380 0 bucket checkpoint complete 2021-09-16T19:43:20.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:20.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:20.795 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:20.843 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:20.844 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:20.861 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.859+0000 7f6fa6809380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:20.861 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.859+0000 7f6fa6809380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:20.861 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.859+0000 7f6fa6809380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:43:20.861 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.859+0000 7f6fa6809380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:20.889 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.888+0000 7f6fa6809380 1 bucket sync caught up with source: 2021-09-16T19:43:20.889 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000006.247.5, , 00000000010.129.5, , 00000000010.89.5, , , , , ] 2021-09-16T19:43:20.889 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000006.247.5, , 00000000010.129.5, , 00000000010.89.5, , , , , ] 2021-09-16T19:43:20.890 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.888+0000 7f6fa6809380 0 bucket checkpoint complete 2021-09-16T19:43:20.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:20.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:20.904 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:20.952 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:20.952 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:20.969 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.968+0000 7f0e9fce2380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:20.970 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.968+0000 7f0e9fce2380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:20.970 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.968+0000 7f0e9fce2380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:43:20.970 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.968+0000 7f0e9fce2380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:20.994 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.992+0000 7f0e9fce2380 1 bucket sync caught up with source: 2021-09-16T19:43:20.994 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000003.247.9, , 00000000003.96.9, , 00000000014.76.5, , , , , ] 2021-09-16T19:43:20.994 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000003.247.9, , 00000000003.96.9, , 00000000014.76.5, , , , , ] 2021-09-16T19:43:20.994 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:20.992+0000 7f0e9fce2380 0 bucket checkpoint complete 2021-09-16T19:43:21.006 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:21.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:21.007 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:21.057 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:21.057 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:21.075 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:21.073+0000 7f0794a3f380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:21.075 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:21.073+0000 7f0794a3f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:21.075 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:21.073+0000 7f0794a3f380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:43:21.075 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:21.073+0000 7f0794a3f380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:21.099 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:21.097+0000 7f0794a3f380 1 bucket sync caught up with source: 2021-09-16T19:43:21.099 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000003.247.9, , 00000000003.96.9, , 00000000014.76.5, , , , , ] 2021-09-16T19:43:21.099 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000003.247.9, , 00000000003.96.9, , 00000000014.76.5, , , , , ] 2021-09-16T19:43:21.099 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:21.097+0000 7f0794a3f380 0 bucket checkpoint complete 2021-09-16T19:43:21.111 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-16 zones={test-zone2, test-zone1} 2021-09-16T19:43:21.116 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:21.119 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:21.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:21.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:21.120 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:21.121 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:21.124 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:21.124 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:21.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:21.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:21.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:21.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:21.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:21.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:21.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:21.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:21.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:21.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:21.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:21.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:21.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:21.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:21.981 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-16 zones={test-zone2, test-zone1} 2021-09-16T19:43:21.982 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-16 zones={test-zone3, test-zone1} 2021-09-16T19:43:21.986 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:21.990 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:21.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:21.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:21.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:21.992 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:21.994 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:21.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:21.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:21.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:22.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:22.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:22.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:22.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:22.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:22.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:22.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:22.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:22.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:22.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:22.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:22.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:23.061 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-16 zones={test-zone3, test-zone1} 2021-09-16T19:43:23.062 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-16 zones={test-zone3, test-zone2} 2021-09-16T19:43:23.066 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:23.069 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:23.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:23.070 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:23.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:23.071 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:23.074 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:23.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:23.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:23.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:23.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:23.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:23.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:23.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:23.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:23.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:23.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:23.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:23.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:23.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:23.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:23.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:23.934 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-16 zones={test-zone3, test-zone2} 2021-09-16T19:43:23.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:23.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:23.935 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:23.991 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:23.992 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:24.011 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.010+0000 7f4c8b1ca380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:24.012 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.010+0000 7f4c8b1ca380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:24.012 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.010+0000 7f4c8b1ca380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:43:24.012 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.010+0000 7f4c8b1ca380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:24.037 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.036+0000 7f4c8b1ca380 1 bucket sync caught up with source: 2021-09-16T19:43:24.038 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000010.26.5, , 00000000003.16.9, , 00000000007.22.9, , , , , ] 2021-09-16T19:43:24.038 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000010.26.5, , 00000000003.16.9, , 00000000007.22.9, , , , , ] 2021-09-16T19:43:24.038 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.036+0000 7f4c8b1ca380 0 bucket checkpoint complete 2021-09-16T19:43:24.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:24.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:24.051 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:24.106 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:24.107 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:24.127 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.125+0000 7f1865260380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:24.127 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.125+0000 7f1865260380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:24.127 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.125+0000 7f1865260380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:43:24.128 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.125+0000 7f1865260380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:24.172 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.170+0000 7f1865260380 1 bucket sync caught up with source: 2021-09-16T19:43:24.172 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000010.26.5, , 00000000003.16.9, , 00000000007.22.9, , , , , ] 2021-09-16T19:43:24.172 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000010.26.5, , 00000000003.16.9, , 00000000007.22.9, , , , , ] 2021-09-16T19:43:24.173 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.170+0000 7f1865260380 0 bucket checkpoint complete 2021-09-16T19:43:24.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:24.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:24.190 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:24.239 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:24.239 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:24.256 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.255+0000 7fb7ba037380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:24.257 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.255+0000 7fb7ba037380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:24.257 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.255+0000 7fb7ba037380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:43:24.257 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.255+0000 7fb7ba037380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:24.280 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.279+0000 7fb7ba037380 1 bucket sync caught up with source: 2021-09-16T19:43:24.281 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000010.26.5, , 00000000003.16.9, , 00000000007.22.9, , , , , ] 2021-09-16T19:43:24.281 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000010.26.5, , 00000000003.16.9, , 00000000007.22.9, , , , , ] 2021-09-16T19:43:24.281 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.279+0000 7fb7ba037380 0 bucket checkpoint complete 2021-09-16T19:43:24.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:24.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:24.293 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:24.343 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:24.343 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:24.360 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.359+0000 7f13fe509380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:24.360 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.359+0000 7f13fe509380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:24.361 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.359+0000 7f13fe509380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:43:24.361 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.359+0000 7f13fe509380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:24.396 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.394+0000 7f13fe509380 1 bucket sync caught up with source: 2021-09-16T19:43:24.396 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000003.178.9, , 00000000010.346.5, , 00000000009.248.9, , , , , ] 2021-09-16T19:43:24.396 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000003.178.9, , 00000000010.346.5, , 00000000009.248.9, , , , , ] 2021-09-16T19:43:24.396 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.394+0000 7f13fe509380 0 bucket checkpoint complete 2021-09-16T19:43:24.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:24.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:24.412 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:24.460 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:24.460 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:24.477 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.476+0000 7fde6eb1f380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:24.478 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.476+0000 7fde6eb1f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:24.478 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.476+0000 7fde6eb1f380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:43:24.478 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.476+0000 7fde6eb1f380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:24.503 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.502+0000 7fde6eb1f380 1 bucket sync caught up with source: 2021-09-16T19:43:24.504 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000003.178.9, , 00000000010.346.5, , 00000000009.248.9, , , , , ] 2021-09-16T19:43:24.504 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000003.178.9, , 00000000010.346.5, , 00000000009.248.9, , , , , ] 2021-09-16T19:43:24.504 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.502+0000 7fde6eb1f380 0 bucket checkpoint complete 2021-09-16T19:43:24.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:24.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:24.517 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:24.565 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:24.565 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:24.582 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.580+0000 7f0ed72b1380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:24.582 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.580+0000 7f0ed72b1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:24.582 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.580+0000 7f0ed72b1380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:43:24.583 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.580+0000 7f0ed72b1380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:24.620 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.619+0000 7f0ed72b1380 1 bucket sync caught up with source: 2021-09-16T19:43:24.621 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000003.187.9, , 00000000003.308.9, , 00000000014.262.5, , , , , ] 2021-09-16T19:43:24.621 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000003.187.9, , 00000000003.308.9, , 00000000014.262.5, , , , , ] 2021-09-16T19:43:24.621 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:24.619+0000 7f0ed72b1380 0 bucket checkpoint complete 2021-09-16T19:43:24.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:24.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:24.639 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:24.689 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:24.689 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:24.707 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.705+0000 7fc600578380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:24.707 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.705+0000 7fc600578380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:24.707 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.705+0000 7fc600578380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:43:24.708 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.705+0000 7fc600578380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:24.731 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.729+0000 7fc600578380 1 bucket sync caught up with source: 2021-09-16T19:43:24.731 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000003.187.9, , 00000000003.308.9, , 00000000014.262.5, , , , , ] 2021-09-16T19:43:24.731 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000003.187.9, , 00000000003.308.9, , 00000000014.262.5, , , , , ] 2021-09-16T19:43:24.731 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:24.729+0000 7fc600578380 0 bucket checkpoint complete 2021-09-16T19:43:24.743 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-17 zones={test-zone2, test-zone1} 2021-09-16T19:43:24.748 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:24.751 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:24.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:24.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:24.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:24.753 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:24.756 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:24.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:24.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:24.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:24.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:24.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:24.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:24.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:24.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:25.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:25.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:25.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:25.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:25.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:25.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:25.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:25.613 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-17 zones={test-zone2, test-zone1} 2021-09-16T19:43:25.614 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-17 zones={test-zone3, test-zone1} 2021-09-16T19:43:25.618 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:25.622 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:25.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:25.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:25.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:25.623 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:25.626 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:25.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:25.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:25.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:25.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:25.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:25.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:25.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:25.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:25.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:25.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:26.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:26.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:26.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:26.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:26.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:26.478 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-17 zones={test-zone3, test-zone1} 2021-09-16T19:43:26.478 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-17 zones={test-zone3, test-zone2} 2021-09-16T19:43:26.483 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:26.486 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:26.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:26.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:26.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:26.488 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:26.491 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:26.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:26.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:26.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:26.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:26.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:26.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:26.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:26.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:26.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:26.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:26.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:27.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:27.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:27.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:27.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:27.406 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-17 zones={test-zone3, test-zone2} 2021-09-16T19:43:27.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:27.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:27.408 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:27.464 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:27.464 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:27.489 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:27.485+0000 7f45c25c4380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:27.490 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:27.485+0000 7f45c25c4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:27.490 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:27.485+0000 7f45c25c4380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:43:27.490 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:27.485+0000 7f45c25c4380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:27.516 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:27.514+0000 7f45c25c4380 1 bucket sync caught up with source: 2021-09-16T19:43:27.516 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000010.27.5, , 00000000003.10.9, , 00000000011.25.5, , , , , ] 2021-09-16T19:43:27.516 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000010.27.5, , 00000000003.10.9, , 00000000011.25.5, , , , , ] 2021-09-16T19:43:27.516 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:27.514+0000 7f45c25c4380 0 bucket checkpoint complete 2021-09-16T19:43:27.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:27.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:27.531 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:27.582 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:27.582 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:27.599 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.598+0000 7fa0f9c5f380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:27.600 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.598+0000 7fa0f9c5f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:27.600 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.598+0000 7fa0f9c5f380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:43:27.600 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.598+0000 7fa0f9c5f380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:27.625 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.623+0000 7fa0f9c5f380 1 bucket sync caught up with source: 2021-09-16T19:43:27.625 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000010.27.5, , 00000000003.10.9, , 00000000011.25.5, , , , , ] 2021-09-16T19:43:27.625 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000010.27.5, , 00000000003.10.9, , 00000000011.25.5, , , , , ] 2021-09-16T19:43:27.626 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.623+0000 7fa0f9c5f380 0 bucket checkpoint complete 2021-09-16T19:43:27.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:27.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:27.638 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:27.688 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:27.689 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:27.707 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:27.706+0000 7f9639c9a380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:27.707 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:27.706+0000 7f9639c9a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:27.708 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:27.706+0000 7f9639c9a380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:43:27.708 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:27.706+0000 7f9639c9a380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:27.731 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:27.729+0000 7f9639c9a380 1 bucket sync caught up with source: 2021-09-16T19:43:27.731 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000010.27.5, , 00000000003.10.9, , 00000000011.25.5, , , , , ] 2021-09-16T19:43:27.731 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000010.27.5, , 00000000003.10.9, , 00000000011.25.5, , , , , ] 2021-09-16T19:43:27.732 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:27.729+0000 7f9639c9a380 0 bucket checkpoint complete 2021-09-16T19:43:27.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:27.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:27.744 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:27.793 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:27.793 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:27.810 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.809+0000 7f23a071a380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:27.811 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.809+0000 7f23a071a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:27.811 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.809+0000 7f23a071a380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:43:27.811 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.809+0000 7f23a071a380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:27.844 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.842+0000 7f23a071a380 1 bucket sync caught up with source: 2021-09-16T19:43:27.844 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000003.168.9, , 00000000010.153.5, , 00000000010.190.5, , , , , ] 2021-09-16T19:43:27.845 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000003.168.9, , 00000000010.153.5, , 00000000010.190.5, , , , , ] 2021-09-16T19:43:27.845 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.842+0000 7f23a071a380 0 bucket checkpoint complete 2021-09-16T19:43:27.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:27.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:27.862 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:27.911 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:27.912 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:27.930 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.928+0000 7fbfe44dd380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:27.930 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.928+0000 7fbfe44dd380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:27.930 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.928+0000 7fbfe44dd380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:43:27.930 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.928+0000 7fbfe44dd380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:27.977 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.976+0000 7fbfe44dd380 1 bucket sync caught up with source: 2021-09-16T19:43:27.977 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000003.168.9, , 00000000010.153.5, , 00000000010.190.5, , , , , ] 2021-09-16T19:43:27.978 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000003.168.9, , 00000000010.153.5, , 00000000010.190.5, , , , , ] 2021-09-16T19:43:27.978 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:27.976+0000 7fbfe44dd380 0 bucket checkpoint complete 2021-09-16T19:43:27.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:27.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:27.997 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:28.045 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:28.046 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:28.062 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:28.061+0000 7f5174477380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:28.063 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:28.061+0000 7f5174477380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:28.063 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:28.061+0000 7f5174477380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:43:28.063 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:28.061+0000 7f5174477380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:28.090 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:28.089+0000 7f5174477380 1 bucket sync caught up with source: 2021-09-16T19:43:28.091 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000003.161.9, , 00000000006.173.5, , 00000000015.119.5, , , , , ] 2021-09-16T19:43:28.091 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000003.161.9, , 00000000006.173.5, , 00000000015.119.5, , , , , ] 2021-09-16T19:43:28.091 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:28.089+0000 7f5174477380 0 bucket checkpoint complete 2021-09-16T19:43:28.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:28.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:28.108 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:28.158 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:28.158 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:28.175 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:28.173+0000 7fc27eee6380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:28.175 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:28.173+0000 7fc27eee6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:28.176 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:28.173+0000 7fc27eee6380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:43:28.176 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:28.173+0000 7fc27eee6380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:28.200 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:28.198+0000 7fc27eee6380 1 bucket sync caught up with source: 2021-09-16T19:43:28.200 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000003.161.9, , 00000000006.173.5, , 00000000015.119.5, , , , , ] 2021-09-16T19:43:28.200 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000003.161.9, , 00000000006.173.5, , 00000000015.119.5, , , , , ] 2021-09-16T19:43:28.201 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:28.198+0000 7fc27eee6380 0 bucket checkpoint complete 2021-09-16T19:43:28.212 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-18 zones={test-zone2, test-zone1} 2021-09-16T19:43:28.218 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:28.221 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:28.221 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:28.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:28.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:28.223 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:28.226 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:28.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:28.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:28.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:28.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:28.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:28.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:28.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:28.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:28.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:28.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:28.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:28.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:28.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:28.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:28.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:29.085 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-18 zones={test-zone2, test-zone1} 2021-09-16T19:43:29.086 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-18 zones={test-zone3, test-zone1} 2021-09-16T19:43:29.091 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:29.094 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:29.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:29.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:29.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:29.096 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:29.143 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:29.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:29.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:29.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:29.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:29.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:29.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:29.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:29.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:29.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:29.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:29.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:29.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:29.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:29.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:29.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:29.994 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-18 zones={test-zone3, test-zone1} 2021-09-16T19:43:29.994 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-18 zones={test-zone3, test-zone2} 2021-09-16T19:43:29.999 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:30.003 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:30.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:30.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:30.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:30.004 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:30.008 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:30.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:30.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:30.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:30.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:30.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:30.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:30.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:30.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:30.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:30.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:30.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:30.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:30.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:30.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:30.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:30.886 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-18 zones={test-zone3, test-zone2} 2021-09-16T19:43:31.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:31.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:31.380 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:31.434 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:31.434 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:31.453 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:31.451+0000 7f77e33ac380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:31.453 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:31.451+0000 7f77e33ac380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:31.454 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:31.451+0000 7f77e33ac380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:43:31.454 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:31.451+0000 7f77e33ac380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:31.479 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:31.477+0000 7f77e33ac380 1 bucket sync caught up with source: 2021-09-16T19:43:31.479 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000011.33.5, , 00000000004.12.5, , 00000000010.21.5, , , , , ] 2021-09-16T19:43:31.479 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000011.33.5, , 00000000004.12.5, , 00000000010.21.5, , , , , ] 2021-09-16T19:43:31.480 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:31.477+0000 7f77e33ac380 0 bucket checkpoint complete 2021-09-16T19:43:31.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:31.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:31.494 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:31.548 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:31.548 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:31.569 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.567+0000 7fba6f98d380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:31.569 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.567+0000 7fba6f98d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:31.569 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.567+0000 7fba6f98d380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:43:31.569 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.567+0000 7fba6f98d380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:31.607 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.606+0000 7fba6f98d380 1 bucket sync caught up with source: 2021-09-16T19:43:31.607 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000011.33.5, , 00000000004.12.5, , 00000000010.21.5, , , , , ] 2021-09-16T19:43:31.608 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000011.33.5, , 00000000004.12.5, , 00000000010.21.5, , , , , ] 2021-09-16T19:43:31.608 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.606+0000 7fba6f98d380 0 bucket checkpoint complete 2021-09-16T19:43:31.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:31.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:31.620 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:31.668 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:31.669 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:31.687 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:31.685+0000 7f5cbdefd380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:31.687 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:31.685+0000 7f5cbdefd380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:31.687 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:31.685+0000 7f5cbdefd380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:43:31.687 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:31.685+0000 7f5cbdefd380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:31.710 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:31.709+0000 7f5cbdefd380 1 bucket sync caught up with source: 2021-09-16T19:43:31.711 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000011.33.5, , 00000000004.12.5, , 00000000010.21.5, , , , , ] 2021-09-16T19:43:31.711 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000011.33.5, , 00000000004.12.5, , 00000000010.21.5, , , , , ] 2021-09-16T19:43:31.711 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:31.709+0000 7f5cbdefd380 0 bucket checkpoint complete 2021-09-16T19:43:31.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:31.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:31.724 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:31.773 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:31.774 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:31.791 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.790+0000 7f09b22ee380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:31.792 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.790+0000 7f09b22ee380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:31.792 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.790+0000 7f09b22ee380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:43:31.792 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.790+0000 7f09b22ee380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:31.821 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.819+0000 7f09b22ee380 1 bucket sync caught up with source: 2021-09-16T19:43:31.821 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000007.250.5, , 00000000011.136.5, , 00000000011.93.5, , , , , ] 2021-09-16T19:43:31.822 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000007.250.5, , 00000000011.136.5, , 00000000011.93.5, , , , , ] 2021-09-16T19:43:31.822 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.819+0000 7f09b22ee380 0 bucket checkpoint complete 2021-09-16T19:43:31.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:31.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:31.839 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:31.889 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:31.889 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:31.906 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.905+0000 7f39a6c7f380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:31.907 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.905+0000 7f39a6c7f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:31.907 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.905+0000 7f39a6c7f380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:43:31.907 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.905+0000 7f39a6c7f380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:31.939 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.937+0000 7f39a6c7f380 1 bucket sync caught up with source: 2021-09-16T19:43:31.939 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000007.250.5, , 00000000011.136.5, , 00000000011.93.5, , , , , ] 2021-09-16T19:43:31.939 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000007.250.5, , 00000000011.136.5, , 00000000011.93.5, , , , , ] 2021-09-16T19:43:31.939 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:31.937+0000 7f39a6c7f380 0 bucket checkpoint complete 2021-09-16T19:43:31.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:31.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:31.957 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:32.004 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:32.005 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:32.022 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:32.020+0000 7f390d0dd380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:32.022 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:32.020+0000 7f390d0dd380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:32.022 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:32.020+0000 7f390d0dd380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:43:32.023 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:32.020+0000 7f390d0dd380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:32.061 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:32.060+0000 7f390d0dd380 1 bucket sync caught up with source: 2021-09-16T19:43:32.062 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000004.250.5, , 00000000004.99.5, , 00000000015.79.5, , , , , ] 2021-09-16T19:43:32.062 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000004.250.5, , 00000000004.99.5, , 00000000015.79.5, , , , , ] 2021-09-16T19:43:32.062 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:32.060+0000 7f390d0dd380 0 bucket checkpoint complete 2021-09-16T19:43:32.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:32.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:32.074 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:32.125 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:32.125 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:32.142 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:32.141+0000 7fca541fc380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:32.143 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:32.141+0000 7fca541fc380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:32.143 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:32.141+0000 7fca541fc380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:43:32.143 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:32.141+0000 7fca541fc380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:32.166 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:32.164+0000 7fca541fc380 1 bucket sync caught up with source: 2021-09-16T19:43:32.166 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000004.250.5, , 00000000004.99.5, , 00000000015.79.5, , , , , ] 2021-09-16T19:43:32.166 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000004.250.5, , 00000000004.99.5, , 00000000015.79.5, , , , , ] 2021-09-16T19:43:32.167 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:32.164+0000 7fca541fc380 0 bucket checkpoint complete 2021-09-16T19:43:32.178 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-16 zones={test-zone2, test-zone1} 2021-09-16T19:43:32.184 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:32.227 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:32.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:32.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:32.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:32.229 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:32.232 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:32.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:32.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:32.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:32.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:32.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:32.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:32.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:32.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:32.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:32.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:32.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:32.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:32.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:32.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:32.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:33.081 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-16 zones={test-zone2, test-zone1} 2021-09-16T19:43:33.082 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-16 zones={test-zone3, test-zone1} 2021-09-16T19:43:33.087 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:33.090 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:33.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:33.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:33.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:33.092 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:33.095 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:33.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:33.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:33.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:33.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:33.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:33.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:33.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:33.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:33.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:33.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:33.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:33.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:33.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:33.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:33.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:33.941 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-16 zones={test-zone3, test-zone1} 2021-09-16T19:43:33.941 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-16 zones={test-zone3, test-zone2} 2021-09-16T19:43:33.946 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:33.949 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:33.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:33.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:33.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:33.951 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:33.995 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:33.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:33.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:33.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:34.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:34.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:34.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:34.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:34.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:34.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:34.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:34.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:34.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:34.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:34.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:34.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:34.918 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-16 zones={test-zone3, test-zone2} 2021-09-16T19:43:34.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:34.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:34.920 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:34.974 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:34.975 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:34.994 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:34.992+0000 7f9c87392380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:34.994 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:34.992+0000 7f9c87392380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:34.995 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:34.992+0000 7f9c87392380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:43:34.995 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:34.992+0000 7f9c87392380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:35.020 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:35.019+0000 7f9c87392380 1 bucket sync caught up with source: 2021-09-16T19:43:35.021 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000011.33.5, , 00000000004.19.5, , 00000000008.25.5, , , , , ] 2021-09-16T19:43:35.021 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000011.33.5, , 00000000004.19.5, , 00000000008.25.5, , , , , ] 2021-09-16T19:43:35.021 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:35.019+0000 7f9c87392380 0 bucket checkpoint complete 2021-09-16T19:43:35.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:35.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:35.034 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:35.087 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:35.087 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:35.106 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.105+0000 7f2375c6f380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:35.107 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.105+0000 7f2375c6f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:35.107 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.105+0000 7f2375c6f380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:43:35.107 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.105+0000 7f2375c6f380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:35.149 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.148+0000 7f2375c6f380 1 bucket sync caught up with source: 2021-09-16T19:43:35.150 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000011.33.5, , 00000000004.19.5, , 00000000008.25.5, , , , , ] 2021-09-16T19:43:35.150 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000011.33.5, , 00000000004.19.5, , 00000000008.25.5, , , , , ] 2021-09-16T19:43:35.150 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.148+0000 7f2375c6f380 0 bucket checkpoint complete 2021-09-16T19:43:35.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:35.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:35.167 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:35.217 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:35.217 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:35.235 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:35.233+0000 7ff6a1828380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:35.235 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:35.233+0000 7ff6a1828380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:35.235 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:35.233+0000 7ff6a1828380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:43:35.236 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:35.233+0000 7ff6a1828380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:35.259 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:35.257+0000 7ff6a1828380 1 bucket sync caught up with source: 2021-09-16T19:43:35.259 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000011.33.5, , 00000000004.19.5, , 00000000008.25.5, , , , , ] 2021-09-16T19:43:35.260 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000011.33.5, , 00000000004.19.5, , 00000000008.25.5, , , , , ] 2021-09-16T19:43:35.260 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:35.257+0000 7ff6a1828380 0 bucket checkpoint complete 2021-09-16T19:43:35.270 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:35.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:35.272 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:35.320 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:35.321 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:35.338 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.336+0000 7fa34156f380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:35.338 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.336+0000 7fa34156f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:35.338 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.336+0000 7fa34156f380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:43:35.339 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.336+0000 7fa34156f380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:35.379 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.377+0000 7fa34156f380 1 bucket sync caught up with source: 2021-09-16T19:43:35.379 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000004.181.5, , 00000000011.353.5, , 00000000010.251.5, , , , , ] 2021-09-16T19:43:35.380 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000004.181.5, , 00000000011.353.5, , 00000000010.251.5, , , , , ] 2021-09-16T19:43:35.380 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.377+0000 7fa34156f380 0 bucket checkpoint complete 2021-09-16T19:43:35.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:35.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:35.399 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:35.447 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:35.448 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:35.465 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.463+0000 7fdd1aa6d380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:35.465 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.463+0000 7fdd1aa6d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:35.465 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.463+0000 7fdd1aa6d380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:43:35.465 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.463+0000 7fdd1aa6d380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:35.498 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.497+0000 7fdd1aa6d380 1 bucket sync caught up with source: 2021-09-16T19:43:35.499 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000004.181.5, , 00000000011.353.5, , 00000000010.251.5, , , , , ] 2021-09-16T19:43:35.499 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000004.181.5, , 00000000011.353.5, , 00000000010.251.5, , , , , ] 2021-09-16T19:43:35.500 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.497+0000 7fdd1aa6d380 0 bucket checkpoint complete 2021-09-16T19:43:35.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:35.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:35.512 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:35.561 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:35.562 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:35.578 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.576+0000 7fd08cebd380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:35.579 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.576+0000 7fd08cebd380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:35.579 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.576+0000 7fd08cebd380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:43:35.579 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.576+0000 7fd08cebd380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:35.616 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.614+0000 7fd08cebd380 1 bucket sync caught up with source: 2021-09-16T19:43:35.616 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000004.190.5, , 00000000004.311.5, , 00000000015.265.5, , , , , ] 2021-09-16T19:43:35.616 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000004.190.5, , 00000000004.311.5, , 00000000015.265.5, , , , , ] 2021-09-16T19:43:35.617 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:35.614+0000 7fd08cebd380 0 bucket checkpoint complete 2021-09-16T19:43:35.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:35.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:35.633 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:35.684 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:35.684 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:35.702 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:35.700+0000 7f6aa273e380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:35.702 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:35.700+0000 7f6aa273e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:35.703 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:35.700+0000 7f6aa273e380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:43:35.703 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:35.700+0000 7f6aa273e380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:35.726 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:35.724+0000 7f6aa273e380 1 bucket sync caught up with source: 2021-09-16T19:43:35.726 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000004.190.5, , 00000000004.311.5, , 00000000015.265.5, , , , , ] 2021-09-16T19:43:35.727 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000004.190.5, , 00000000004.311.5, , 00000000015.265.5, , , , , ] 2021-09-16T19:43:35.727 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:35.724+0000 7f6aa273e380 0 bucket checkpoint complete 2021-09-16T19:43:35.738 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-17 zones={test-zone2, test-zone1} 2021-09-16T19:43:35.743 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:35.747 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:35.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:35.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:35.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:35.748 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:35.751 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:35.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:35.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:35.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:35.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:35.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:35.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:35.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:35.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:36.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:36.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:36.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:36.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:36.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:36.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:36.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:36.626 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-17 zones={test-zone2, test-zone1} 2021-09-16T19:43:36.626 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-17 zones={test-zone3, test-zone1} 2021-09-16T19:43:36.631 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:36.634 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:36.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:36.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:36.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:36.635 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:36.639 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:36.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:36.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:36.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:36.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:36.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:36.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:36.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:36.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:36.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:37.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:37.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:37.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:37.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:37.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:37.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:37.487 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-17 zones={test-zone3, test-zone1} 2021-09-16T19:43:37.488 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-17 zones={test-zone3, test-zone2} 2021-09-16T19:43:37.494 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:37.504 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:37.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:37.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:37.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:37.506 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:37.510 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:37.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:37.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:37.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:37.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:37.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:37.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:37.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:37.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:37.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:37.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:37.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:38.039 INFO:tasks.rgw.c1.client.1.smithi109.stdout:2021-09-16T19:43:38.038+0000 7f2506e7c700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-16T19:43:38.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:38.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:38.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:38.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:38.454 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-17 zones={test-zone3, test-zone2} 2021-09-16T19:43:38.455 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:38.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:38.456 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:38.514 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:38.514 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:38.540 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:38.538+0000 7f0522ad9380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:38.540 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:38.538+0000 7f0522ad9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:38.541 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:38.538+0000 7f0522ad9380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:43:38.541 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:38.538+0000 7f0522ad9380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:38.568 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:38.566+0000 7f0522ad9380 1 bucket sync caught up with source: 2021-09-16T19:43:38.568 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000011.34.5, , 00000000004.13.5, , 00000000012.28.5, , , , , ] 2021-09-16T19:43:38.569 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000011.34.5, , 00000000004.13.5, , 00000000012.28.5, , , , , ] 2021-09-16T19:43:38.569 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:38.566+0000 7f0522ad9380 0 bucket checkpoint complete 2021-09-16T19:43:38.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:38.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:38.583 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:38.635 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:38.635 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:38.653 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:38.653+0000 7f76295df380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:38.654 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:38.653+0000 7f76295df380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:38.654 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:38.653+0000 7f76295df380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:43:38.654 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:38.653+0000 7f76295df380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:38.678 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:38.678+0000 7f76295df380 1 bucket sync caught up with source: 2021-09-16T19:43:38.679 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000011.34.5, , 00000000004.13.5, , 00000000012.28.5, , , , , ] 2021-09-16T19:43:38.679 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000011.34.5, , 00000000004.13.5, , 00000000012.28.5, , , , , ] 2021-09-16T19:43:38.679 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:38.678+0000 7f76295df380 0 bucket checkpoint complete 2021-09-16T19:43:38.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:38.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:38.692 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:38.743 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:38.744 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:38.761 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:38.760+0000 7f49ac6ee380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:38.762 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:38.760+0000 7f49ac6ee380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:38.762 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:38.760+0000 7f49ac6ee380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:43:38.762 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:38.760+0000 7f49ac6ee380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:38.785 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:38.783+0000 7f49ac6ee380 1 bucket sync caught up with source: 2021-09-16T19:43:38.785 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000011.34.5, , 00000000004.13.5, , 00000000012.28.5, , , , , ] 2021-09-16T19:43:38.786 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000011.34.5, , 00000000004.13.5, , 00000000012.28.5, , , , , ] 2021-09-16T19:43:38.786 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:38.783+0000 7f49ac6ee380 0 bucket checkpoint complete 2021-09-16T19:43:38.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:38.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:38.798 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:38.848 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:38.849 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:38.867 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:38.866+0000 7f3387e0b380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:38.867 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:38.866+0000 7f3387e0b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:38.867 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:38.866+0000 7f3387e0b380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:43:38.867 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:38.866+0000 7f3387e0b380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:38.909 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:38.909+0000 7f3387e0b380 1 bucket sync caught up with source: 2021-09-16T19:43:38.910 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000004.171.5, , 00000000011.159.5, , 00000000011.194.5, , , , , ] 2021-09-16T19:43:38.910 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000004.171.5, , 00000000011.159.5, , 00000000011.194.5, , , , , ] 2021-09-16T19:43:38.910 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:38.909+0000 7f3387e0b380 0 bucket checkpoint complete 2021-09-16T19:43:38.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:38.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:38.930 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:38.979 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:38.979 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:38.996 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:38.996+0000 7fcaba882380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:38.997 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:38.996+0000 7fcaba882380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:38.997 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:38.996+0000 7fcaba882380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:43:38.997 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:38.996+0000 7fcaba882380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:39.038 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:39.038+0000 7fcaba882380 1 bucket sync caught up with source: 2021-09-16T19:43:39.039 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000004.171.5, , 00000000011.159.5, , 00000000011.194.5, , , , , ] 2021-09-16T19:43:39.039 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000004.171.5, , 00000000011.159.5, , 00000000011.194.5, , , , , ] 2021-09-16T19:43:39.039 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:39.038+0000 7fcaba882380 0 bucket checkpoint complete 2021-09-16T19:43:39.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:39.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:39.060 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:39.108 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:39.108 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:39.126 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:39.125+0000 7f19b08fa380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:39.126 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:39.125+0000 7f19b08fa380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:39.126 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:39.125+0000 7f19b08fa380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:43:39.126 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:39.125+0000 7f19b08fa380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:39.155 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:39.155+0000 7f19b08fa380 1 bucket sync caught up with source: 2021-09-16T19:43:39.155 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, 00000000004.164.5, , 00000000007.177.5, , 00000000016.122.5, , , , , ] 2021-09-16T19:43:39.156 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, 00000000004.164.5, , 00000000007.177.5, , 00000000016.122.5, , , , , ] 2021-09-16T19:43:39.156 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:39.155+0000 7f19b08fa380 0 bucket checkpoint complete 2021-09-16T19:43:39.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:39.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:39.168 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:39.218 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:39.218 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:39.237 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:39.235+0000 7f88d3e4b380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:39.237 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:39.235+0000 7f88d3e4b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:39.237 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:39.235+0000 7f88d3e4b380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:43:39.237 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:39.235+0000 7f88d3e4b380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:39.260 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:39.259+0000 7f88d3e4b380 1 bucket sync caught up with source: 2021-09-16T19:43:39.260 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, 00000000004.164.5, , 00000000007.177.5, , 00000000016.122.5, , , , , ] 2021-09-16T19:43:39.261 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, 00000000004.164.5, , 00000000007.177.5, , 00000000016.122.5, , , , , ] 2021-09-16T19:43:39.261 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:39.259+0000 7f88d3e4b380 0 bucket checkpoint complete 2021-09-16T19:43:39.273 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-18 zones={test-zone2, test-zone1} 2021-09-16T19:43:39.278 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:39.281 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:39.281 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:39.282 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:39.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:39.283 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:39.286 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:39.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:39.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:39.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:39.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:39.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:39.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:39.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:39.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:39.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:39.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:39.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:39.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:39.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:39.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:40.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:40.145 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-18 zones={test-zone2, test-zone1} 2021-09-16T19:43:40.146 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-18 zones={test-zone3, test-zone1} 2021-09-16T19:43:40.150 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:40.153 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:40.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:40.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:40.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:40.155 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:40.158 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:40.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:40.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:40.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:40.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:40.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:40.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:40.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:40.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:40.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:40.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:40.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:40.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:40.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:40.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:40.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:41.018 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-18 zones={test-zone3, test-zone1} 2021-09-16T19:43:41.018 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-18 zones={test-zone3, test-zone2} 2021-09-16T19:43:41.024 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:41.028 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:41.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:41.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:41.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:41.029 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:41.075 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:41.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-16T19:43:41.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-16T19:43:41.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-16T19:43:41.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:41.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:41.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:41.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:41.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:41.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-16T19:43:41.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:41.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:41.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-16T19:43:41.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:41.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:41.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-16T19:43:41.995 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-18 zones={test-zone3, test-zone2} 2021-09-16T19:43:41.996 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2021-09-16T19:43:42.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:43:42.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:43:42.029 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:43:42.071 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:42.072 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:42.129 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:43:42.144 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:43:42.145 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:43:42.145 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:43:42.146 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:43:42.146 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:43:42.146 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:43:42.146 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:43:42.147 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:43:42.147 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:43:42.147 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:43:42.147 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821422.027398_129.1", 2021-09-16T19:43:42.147 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:43:42.027398Z" 2021-09-16T19:43:42.148 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:43:42.148 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:43:42.148 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:43:42.148 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:17.633408Z" 2021-09-16T19:43:42.149 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:43:42.149 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:43:42.150 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:43:42.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:43:42.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:43:42.152 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:43:42.201 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:42.201 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:42.254 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:43:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:43:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:43:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:43:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:43:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:43:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:43:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:43:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:43:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:43:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:43:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:43:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:43:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:43:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:43:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:43:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821422.005586_124.1", 2021-09-16T19:43:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:43:42.264 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:42.264 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:43:42.005586Z", 2021-09-16T19:43:42.264 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:42.264 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:42.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:42.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:42.265 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:43:42.265 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:42.265 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:42.266 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:43:42.266 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:42.266 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:43:42.266 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:42.267 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:43:42.267 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:42.267 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:42.267 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:43:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:43:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:43:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:43:42.269 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:42.269 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:43:42.270 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821422.027398_129.1 target=1_1631821422.005586_124.1 2021-09-16T19:43:47.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:43:47.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:43:47.273 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:43:47.323 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:47.324 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:47.377 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:43:47.377 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:43:47.377 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:43:47.378 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:43:47.378 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:43:47.378 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:43:47.378 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:43:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:43:47.379 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:47.380 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:47.380 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:43:47.380 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:47.380 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:43:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:43:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:47.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:43:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:47.382 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:43:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:43:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:47.383 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:43:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:47.384 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:43:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821422.027398_129.1", 2021-09-16T19:43:47.385 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:43:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:43:42.027398Z", 2021-09-16T19:43:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:47.386 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:43:47.387 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:43:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:43:47.388 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:43:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:47.389 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:43:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:43:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:43:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:43:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:47.391 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:43:47.392 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:43:47.393 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:43:47.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:43:47.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:43:47.394 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:43:47.443 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:47.443 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:47.519 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:43:47.520 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:43:47.520 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:43:47.520 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:43:47.520 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:43:47.521 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:43:47.521 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:43:47.521 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:43:47.521 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:43:47.521 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:43:47.522 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:43:47.522 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:43:47.522 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:43:47.522 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:43:47.523 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:43:47.523 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:43:47.523 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:43:47.523 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:43:47.523 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:43:47.524 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:43:47.524 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:43:47.524 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:43:47.524 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:43:47.524 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:43:47.525 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:43:47.525 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:43:47.525 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:43:47.525 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:43:47.526 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:43:47.526 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:43:47.526 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:43:47.526 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:43:47.526 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:43:47.527 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:43:47.527 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:43:47.527 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:43:47.527 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:43:47.528 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821422.027398_129.1", 2021-09-16T19:43:47.528 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:43:47.528 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:43:47.528 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:43:47.528 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:43:42.027398Z", 2021-09-16T19:43:47.529 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:43:47.529 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:43:47.529 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:43:47.529 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:43:47.529 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:43:47.530 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:43:47.530 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:43:47.530 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:43:47.530 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:43:47.531 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:43:47.531 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:43:47.531 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:43:47.531 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:43:47.531 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:43:47.532 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:43:47.532 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:43:47.532 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:43:47.532 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:43:47.532 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:43:47.533 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:43:47.533 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:43:47.533 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:43:47.534 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-16T19:43:47.534 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-16T19:43:47.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:43:47.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:43:47.536 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:43:47.582 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:47.582 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:47.631 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:43:47.632 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:43:47.632 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:43:47.632 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:43:47.633 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:43:47.633 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:43:47.633 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:47.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:47.633 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:43:47.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:47.634 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:43:47.634 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:47.634 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:47.635 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:43:47.635 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:47.635 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:43:47.635 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:47.636 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:43:47.636 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:47.636 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:47.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:47.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:47.637 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:43:47.637 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:47.637 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:47.637 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:43:47.637 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:47.638 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:43:47.638 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:47.638 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:43:47.638 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:47.639 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:47.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:47.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:47.639 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:43:47.640 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:47.640 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:47.640 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821422.027398_129.1", 2021-09-16T19:43:47.640 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:47.640 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:43:47.641 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:47.641 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:43:42.027398Z", 2021-09-16T19:43:47.641 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:47.641 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:47.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:47.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:43:47.642 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:43:47.642 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:43:47.642 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:43:47.643 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:43:47.643 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:43:47.643 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:43:47.643 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:43:47.644 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:43:47.644 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:43:47.644 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:47.644 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:47.644 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:43:47.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:43:47.645 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:43:47.645 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:43:47.645 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:43:47.645 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:43:47.646 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:43:47.647 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-16T19:43:47.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:47.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:47.696 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:47.746 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:47.746 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:47.765 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:47.763+0000 7f01f76fa380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:47.765 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:47.763+0000 7f01f76fa380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:47.765 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:47.763+0000 7f01f76fa380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:43:47.765 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:47.763+0000 7f01f76fa380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:47.789 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:47.787+0000 7f01f76fa380 1 waiting for incremental sync to catch up: 2021-09-16T19:43:47.789 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , , , ] 2021-09-16T19:43:47.789 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , 00000000002.24.10, , , , , ] 2021-09-16T19:43:52.790 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:52.788+0000 7f01f76fa380 1 bucket sync caught up with source: 2021-09-16T19:43:52.791 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , 00000000006.32.9, , , , , ] 2021-09-16T19:43:52.791 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , 00000000002.24.10, , , , , ] 2021-09-16T19:43:52.791 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:52.788+0000 7f01f76fa380 0 bucket checkpoint complete 2021-09-16T19:43:52.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:52.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:52.808 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:52.862 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:52.862 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:52.881 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:52.880+0000 7f0e0ab62380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:52.882 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:52.880+0000 7f0e0ab62380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:52.882 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:52.880+0000 7f0e0ab62380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:43:52.882 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:52.880+0000 7f0e0ab62380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:52.910 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:52.909+0000 7f0e0ab62380 1 bucket sync caught up with source: 2021-09-16T19:43:52.910 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , 00000000006.32.9, , , , , ] 2021-09-16T19:43:52.911 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , 00000000006.32.9, , , , , ] 2021-09-16T19:43:52.911 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:52.909+0000 7f0e0ab62380 0 bucket checkpoint complete 2021-09-16T19:43:52.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:52.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:52.935 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:52.991 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:52.991 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:53.015 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:53.014+0000 7f82ac6f5380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:53.015 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:53.014+0000 7f82ac6f5380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:53.015 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:53.014+0000 7f82ac6f5380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:43:53.016 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:53.014+0000 7f82ac6f5380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:53.042 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:53.042+0000 7f82ac6f5380 1 bucket sync caught up with source: 2021-09-16T19:43:53.081 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , 00000000006.32.9, , , , , ] 2021-09-16T19:43:53.082 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , 00000000006.32.9, , , , , ] 2021-09-16T19:43:53.082 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:53.042+0000 7f82ac6f5380 0 bucket checkpoint complete 2021-09-16T19:43:53.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:53.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:53.084 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:53.133 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:53.133 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:53.151 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.149+0000 7f9b7b00e380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:53.151 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.149+0000 7f9b7b00e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:53.151 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.149+0000 7f9b7b00e380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:43:53.152 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.149+0000 7f9b7b00e380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:53.196 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.194+0000 7f9b7b00e380 1 bucket sync caught up with source: 2021-09-16T19:43:53.196 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , 00000000004.400.9, , , , , ] 2021-09-16T19:43:53.196 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , 00000000004.400.9, , , , , ] 2021-09-16T19:43:53.197 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.194+0000 7f9b7b00e380 0 bucket checkpoint complete 2021-09-16T19:43:53.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:53.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:53.211 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:53.259 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:53.260 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:53.277 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.275+0000 7f8b52534380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:53.277 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.275+0000 7f8b52534380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:53.277 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.275+0000 7f8b52534380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:43:53.277 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.275+0000 7f8b52534380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:53.305 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.303+0000 7f8b52534380 1 bucket sync caught up with source: 2021-09-16T19:43:53.305 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , 00000000004.400.9, , , , , ] 2021-09-16T19:43:53.306 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , 00000000004.400.9, , , , , ] 2021-09-16T19:43:53.306 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.303+0000 7f8b52534380 0 bucket checkpoint complete 2021-09-16T19:43:53.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:53.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:53.321 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:53.369 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:53.370 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:53.387 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.386+0000 7fdf8ffca380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:53.387 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.386+0000 7fdf8ffca380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:53.387 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.386+0000 7fdf8ffca380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:43:53.388 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.386+0000 7fdf8ffca380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:53.423 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.422+0000 7fdf8ffca380 1 bucket sync caught up with source: 2021-09-16T19:43:53.424 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , 00000000004.378.9, , , , , ] 2021-09-16T19:43:53.424 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , 00000000004.378.9, , , , , ] 2021-09-16T19:43:53.424 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:53.422+0000 7fdf8ffca380 0 bucket checkpoint complete 2021-09-16T19:43:53.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:53.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:53.442 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:53.498 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:53.498 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:53.518 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:53.517+0000 7f28bcc8a380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:53.518 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:53.517+0000 7f28bcc8a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:53.518 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:53.517+0000 7f28bcc8a380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:43:53.519 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:53.517+0000 7f28bcc8a380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:53.545 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:53.544+0000 7f28bcc8a380 1 bucket sync caught up with source: 2021-09-16T19:43:53.545 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , 00000000004.378.9, , , , , ] 2021-09-16T19:43:53.545 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , 00000000004.378.9, , , , , ] 2021-09-16T19:43:53.546 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:53.544+0000 7f28bcc8a380 0 bucket checkpoint complete 2021-09-16T19:43:53.558 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-19 zones={test-zone2, test-zone1} 2021-09-16T19:43:53.563 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:53.607 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:53.608 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:53.610 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:53.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:53.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:53.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:53.773 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-19 zones={test-zone2, test-zone1} 2021-09-16T19:43:53.774 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-19 zones={test-zone3, test-zone1} 2021-09-16T19:43:53.779 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:53.823 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:53.823 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:53.827 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:53.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:53.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:53.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:53.985 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-19 zones={test-zone3, test-zone1} 2021-09-16T19:43:53.986 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-19 zones={test-zone3, test-zone2} 2021-09-16T19:43:53.990 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:43:54.035 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:54.036 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:43:54.039 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:43:54.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:54.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:54.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:43:54.242 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-19 zones={test-zone3, test-zone2} 2021-09-16T19:43:55.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:55.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:55.625 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:55.674 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:55.674 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:55.692 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:55.691+0000 7f4e5e206380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:55.692 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:55.691+0000 7f4e5e206380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:55.693 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:55.691+0000 7f4e5e206380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:43:55.693 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:55.691+0000 7f4e5e206380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:55.717 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:55.717+0000 7f4e5e206380 1 bucket sync caught up with source: 2021-09-16T19:43:55.718 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , 00000000006.32.9, , , 00000000052.133.11, , ] 2021-09-16T19:43:55.718 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , 00000000006.32.9, , , 00000000052.133.11, , ] 2021-09-16T19:43:55.718 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:43:55.717+0000 7f4e5e206380 0 bucket checkpoint complete 2021-09-16T19:43:55.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:55.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:43:55.732 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:43:55.785 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:43:55.786 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:43:55.802 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:55.801+0000 7f838f205380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:43:55.803 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:55.801+0000 7f838f205380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:43:55.803 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:55.801+0000 7f838f205380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:43:55.803 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:55.801+0000 7f838f205380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:43:55.841 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:43:55.839+0000 7f838f205380 1 waiting for incremental sync to catch up: 2021-09-16T19:43:55.841 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , 00000000006.32.9, , , 00000000054.137.11, , ] 2021-09-16T19:43:55.842 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , 00000000006.32.9, , , 00000000055.139.9, , ] 2021-09-16T19:43:57.983 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2021-09-16T19:43:57.982+0000 7f0df1560700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-16T19:44:00.842 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:00.841+0000 7f838f205380 1 bucket sync caught up with source: 2021-09-16T19:44:00.843 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , 00000000006.32.9, , , 00000000056.141.9, , ] 2021-09-16T19:44:00.843 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , 00000000006.32.9, , , 00000000055.139.9, , ] 2021-09-16T19:44:00.843 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:00.841+0000 7f838f205380 0 bucket checkpoint complete 2021-09-16T19:44:00.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:00.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:00.860 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:00.915 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:00.915 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:00.935 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:00.935+0000 7f7efe060380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:00.936 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:00.935+0000 7f7efe060380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:00.936 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:00.935+0000 7f7efe060380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:44:00.936 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:00.935+0000 7f7efe060380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:00.961 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:00.960+0000 7f7efe060380 1 bucket sync caught up with source: 2021-09-16T19:44:00.961 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , 00000000006.32.9, , , 00000000056.141.9, , ] 2021-09-16T19:44:00.962 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , 00000000006.32.9, , , 00000000056.141.9, , ] 2021-09-16T19:44:00.962 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:00.960+0000 7f7efe060380 0 bucket checkpoint complete 2021-09-16T19:44:00.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:00.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:00.975 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:01.031 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:01.032 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:01.052 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.050+0000 7f4b6102e380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:01.052 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.050+0000 7f4b6102e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:01.052 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.050+0000 7f4b6102e380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:44:01.052 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.050+0000 7f4b6102e380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:01.082 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.081+0000 7f4b6102e380 1 bucket sync caught up with source: 2021-09-16T19:44:01.083 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , 00000000004.400.9, , , 00000000057.329.9, , ] 2021-09-16T19:44:01.083 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , 00000000004.400.9, , , 00000000057.329.9, , ] 2021-09-16T19:44:01.083 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.081+0000 7f4b6102e380 0 bucket checkpoint complete 2021-09-16T19:44:01.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:01.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:01.101 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:01.150 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:01.150 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:01.167 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.166+0000 7f67ef975380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:01.167 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.166+0000 7f67ef975380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:01.168 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.166+0000 7f67ef975380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:44:01.168 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.166+0000 7f67ef975380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:01.219 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.217+0000 7f67ef975380 1 bucket sync caught up with source: 2021-09-16T19:44:01.219 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , 00000000004.400.9, , , 00000000057.329.9, , ] 2021-09-16T19:44:01.219 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , 00000000004.400.9, , , 00000000057.329.9, , ] 2021-09-16T19:44:01.219 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.217+0000 7f67ef975380 0 bucket checkpoint complete 2021-09-16T19:44:01.232 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:01.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:01.234 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:01.281 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:01.282 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:01.299 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.297+0000 7fe2412f8380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:01.299 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.297+0000 7fe2412f8380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:01.299 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.297+0000 7fe2412f8380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:44:01.299 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.297+0000 7fe2412f8380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:01.334 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.333+0000 7fe2412f8380 1 bucket sync caught up with source: 2021-09-16T19:44:01.334 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , 00000000004.378.9, , , 00000000052.376.9, , ] 2021-09-16T19:44:01.335 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , 00000000004.378.9, , , 00000000052.376.9, , ] 2021-09-16T19:44:01.335 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:01.333+0000 7fe2412f8380 0 bucket checkpoint complete 2021-09-16T19:44:01.348 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:01.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:01.349 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:01.403 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:01.403 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:01.422 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:01.422+0000 7fae1bfcc380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:01.422 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:01.422+0000 7fae1bfcc380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:01.423 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:01.422+0000 7fae1bfcc380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:44:01.423 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:01.422+0000 7fae1bfcc380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:01.448 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:01.447+0000 7fae1bfcc380 1 bucket sync caught up with source: 2021-09-16T19:44:01.448 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , 00000000004.378.9, , , 00000000052.376.9, , ] 2021-09-16T19:44:01.448 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , 00000000004.378.9, , , 00000000052.376.9, , ] 2021-09-16T19:44:01.449 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:01.447+0000 7fae1bfcc380 0 bucket checkpoint complete 2021-09-16T19:44:01.461 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-19 zones={test-zone2, test-zone1} 2021-09-16T19:44:01.466 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:44:01.471 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:44:01.471 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.484 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:44:01.489 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:44:01.489 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.490 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.497 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.497 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.498 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.498 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:01.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:44:01.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:44:01.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:44:01.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:01.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:01.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:01.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:01.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:02.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:02.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:02.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:02.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:02.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:02.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:02.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:02.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:02.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:02.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:02.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:02.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:03.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:03.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:03.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:03.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:03.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:03.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:03.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:03.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:03.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:03.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:03.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:03.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:04.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:04.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:04.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:04.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:04.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:04.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:04.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:04.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:04.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:04.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:04.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:04.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:05.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:05.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:05.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:05.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:05.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:05.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:05.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:05.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:05.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:05.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:05.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:06.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:06.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:06.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:06.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:06.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:06.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:06.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:06.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:06.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:06.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:06.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:07.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:07.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:07.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:07.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:07.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:07.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:07.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:07.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:07.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:07.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:07.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:07.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:08.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:08.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:08.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:08.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:08.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:08.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:08.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:08.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:08.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:08.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:08.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:08.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:09.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:09.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:09.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:09.343 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-19 zones={test-zone2, test-zone1} 2021-09-16T19:44:09.344 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-19 zones={test-zone3, test-zone1} 2021-09-16T19:44:09.350 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:44:09.358 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:44:09.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.371 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.372 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.373 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.376 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.377 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.377 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.378 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.380 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:44:09.386 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:44:09.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.399 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:09.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:44:09.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:44:09.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:44:09.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:09.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:09.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:09.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:09.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:09.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:10.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:10.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:10.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:10.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:10.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:10.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:10.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:10.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:10.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:10.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:10.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:10.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:11.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:11.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:11.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:11.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:11.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:11.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:11.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:11.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:11.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:11.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:11.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:11.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:12.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:12.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:12.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:12.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:12.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:12.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:12.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:12.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:12.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:12.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:12.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:13.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:13.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:13.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:13.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:13.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:13.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:13.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:13.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:13.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:13.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:13.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:13.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:14.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:14.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:14.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:14.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:14.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:14.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:14.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:14.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:14.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:14.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:14.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:14.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:15.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:15.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:15.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:15.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:15.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:15.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:15.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:15.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:15.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:15.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:15.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:16.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:16.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:16.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:16.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:16.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:16.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:16.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:16.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:16.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:16.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:16.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:16.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:17.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:17.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:17.238 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-19 zones={test-zone3, test-zone1} 2021-09-16T19:44:17.239 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-19 zones={test-zone3, test-zone2} 2021-09-16T19:44:17.244 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:44:17.292 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:44:17.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.300 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.306 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:44:17.311 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-16T19:44:17.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.312 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.314 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.315 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.318 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.318 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.318 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:17.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:44:17.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:44:17.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-16T19:44:17.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:17.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:17.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:17.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:17.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:17.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:17.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:18.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:18.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:18.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:18.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:18.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:18.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:18.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:18.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:18.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:18.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:19.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:19.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:19.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:19.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:19.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:19.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:19.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:19.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:19.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:19.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:20.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:20.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:20.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:20.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:20.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:20.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:20.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:20.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:20.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:20.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:20.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:20.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:21.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:21.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:21.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:21.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:21.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:21.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:21.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:21.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:21.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:21.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:21.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:21.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:22.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:22.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:22.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:22.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:22.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:22.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:22.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:22.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:22.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:22.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:22.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:23.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:23.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:23.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:23.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:23.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:23.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:23.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:23.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:23.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:23.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:23.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:23.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:24.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:24.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:24.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:24.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:24.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:24.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:24.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:24.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:24.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:24.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:24.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:24.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:25.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:25.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:25.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:25.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:25.454 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-19 zones={test-zone3, test-zone2} 2021-09-16T19:44:25.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:25.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:25.456 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:25.504 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:25.504 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:25.572 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:44:25.573 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:25.573 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-16T19:44:25.573 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:43:38.784578Z" 2021-09-16T19:44:25.574 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:25.574 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:25.574 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-16T19:44:25.574 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:44:00.787139Z" 2021-09-16T19:44:25.575 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:25.575 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:25.575 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-16T19:44:25.575 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:44:00.788650Z" 2021-09-16T19:44:25.575 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:25.576 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:25.576 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:44:25.576 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:44:25.576 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:25.577 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:44:25.578 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-09-16T19:44:25.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:25.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:25.579 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:25.631 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:25.632 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:25.688 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:44:25.688 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:44:25.688 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:44:25.689 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:44:25.689 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:44:25.689 INFO:teuthology.orchestra.run.smithi118.stdout: "instance_id": 6785307543928147341 2021-09-16T19:44:25.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:25.689 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:44:25.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:25.690 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:44:25.690 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:25.690 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "incremental-sync", 2021-09-16T19:44:25.691 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-16T19:44:25.691 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:25.691 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:25.691 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:25.691 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:43:38.784578Z" 2021-09-16T19:44:25.692 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:25.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:25.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:25.692 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:44:25.693 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:25.693 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "incremental-sync", 2021-09-16T19:44:25.693 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-16T19:44:25.693 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:25.693 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:25.694 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:25.694 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:00.787139Z" 2021-09-16T19:44:25.694 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:25.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:25.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:25.695 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:44:25.695 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:25.695 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "incremental-sync", 2021-09-16T19:44:25.695 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-16T19:44:25.695 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:25.696 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:25.696 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:25.696 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:00.788650Z" 2021-09-16T19:44:25.696 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:25.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:25.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:25.697 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:44:25.697 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:25.697 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "incremental-sync", 2021-09-16T19:44:25.698 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:44:25.698 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:25.698 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:25.698 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:25.698 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000" 2021-09-16T19:44:25.699 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:25.699 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:25.699 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:44:25.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:25.699 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:44:25.700 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 0, 2021-09-16T19:44:25.700 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 0 2021-09-16T19:44:25.700 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:25.700 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:44:25.702 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-09-16T19:44:25.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:25.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:25.703 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:25.746 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:25.747 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:25.818 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:44:25.818 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:25.819 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-16T19:44:25.819 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:43:38.784578Z" 2021-09-16T19:44:25.819 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:25.819 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:25.820 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-16T19:44:25.820 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:44:00.787139Z" 2021-09-16T19:44:25.820 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:25.820 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:25.820 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-16T19:44:25.821 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:44:00.788650Z" 2021-09-16T19:44:25.821 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:25.821 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:25.821 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:44:25.822 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:44:25.822 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:25.822 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:44:25.823 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-09-16T19:44:25.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:25.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:25.825 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:25.908 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:25.908 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:25.983 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:44:25.984 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:44:25.984 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:44:25.984 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:44:25.984 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:44:25.985 INFO:teuthology.orchestra.run.smithi109.stdout: "instance_id": 8611250324479679307 2021-09-16T19:44:25.985 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:25.985 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:44:25.985 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:25.986 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:44:25.986 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:25.986 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "incremental-sync", 2021-09-16T19:44:25.986 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-16T19:44:25.987 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:25.987 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:25.987 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:25.987 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:43:38.784578Z" 2021-09-16T19:44:25.988 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:25.988 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:25.988 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:25.988 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:44:25.989 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:25.989 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "incremental-sync", 2021-09-16T19:44:25.989 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-16T19:44:25.989 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:25.990 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:25.990 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:25.990 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:44:00.787139Z" 2021-09-16T19:44:25.990 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:25.991 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:25.991 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:25.991 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:44:25.991 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:25.991 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "incremental-sync", 2021-09-16T19:44:25.992 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-16T19:44:25.992 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:25.992 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:25.992 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:25.993 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:44:00.788650Z" 2021-09-16T19:44:25.993 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:25.993 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:25.993 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:25.994 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:44:25.994 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:25.994 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "incremental-sync", 2021-09-16T19:44:25.994 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:44:25.995 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:25.995 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:25.995 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:25.996 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000" 2021-09-16T19:44:25.996 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:25.996 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:25.996 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:44:25.996 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:25.997 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:44:25.997 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 0, 2021-09-16T19:44:25.997 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 0 2021-09-16T19:44:25.997 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:25.998 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:44:25.999 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-09-16T19:44:26.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.001 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:26.042 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:26.042 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:26.116 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:44:26.117 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.117 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-16T19:44:26.117 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:43:38.784578Z" 2021-09-16T19:44:26.118 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.118 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.118 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-16T19:44:26.118 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:44:00.787139Z" 2021-09-16T19:44:26.119 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.119 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.119 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-16T19:44:26.119 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:44:00.788650Z" 2021-09-16T19:44:26.119 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.120 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.120 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:44:26.120 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:44:26.120 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.121 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:44:26.122 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-09-16T19:44:26.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.124 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:26.175 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:26.175 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:26.234 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:44:26.234 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:44:26.234 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:44:26.235 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:44:26.235 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:44:26.235 INFO:teuthology.orchestra.run.smithi118.stdout: "instance_id": 15366304219181685110 2021-09-16T19:44:26.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:26.236 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:44:26.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:26.236 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:44:26.236 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:26.236 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "incremental-sync", 2021-09-16T19:44:26.237 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-16T19:44:26.237 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:26.237 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:26.237 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:26.238 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:43:38.784578Z" 2021-09-16T19:44:26.238 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:26.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:26.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:26.239 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:44:26.239 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:26.239 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "incremental-sync", 2021-09-16T19:44:26.239 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-16T19:44:26.239 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:26.240 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:26.240 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:26.240 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:00.787139Z" 2021-09-16T19:44:26.240 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:26.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:26.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:26.241 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:44:26.241 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:26.242 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "incremental-sync", 2021-09-16T19:44:26.242 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-16T19:44:26.242 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:26.242 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:26.243 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:26.243 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:00.788650Z" 2021-09-16T19:44:26.243 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:26.243 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:26.243 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:26.244 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:44:26.244 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:26.244 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "incremental-sync", 2021-09-16T19:44:26.244 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:44:26.245 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:26.245 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:26.245 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:26.245 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000" 2021-09-16T19:44:26.245 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:26.246 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:26.246 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:44:26.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:26.247 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:44:26.247 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 0, 2021-09-16T19:44:26.247 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 0 2021-09-16T19:44:26.247 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:26.247 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:44:26.249 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-09-16T19:44:26.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.250 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:26.293 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:26.294 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:26.348 INFO:teuthology.orchestra.run.smithi118.stdout:[ 2021-09-16T19:44:26.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:26.348 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-16T19:44:26.348 INFO:teuthology.orchestra.run.smithi118.stdout: "last_update": "2021-09-16T19:43:38.922787Z" 2021-09-16T19:44:26.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:26.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:26.349 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-16T19:44:26.349 INFO:teuthology.orchestra.run.smithi118.stdout: "last_update": "2021-09-16T19:44:00.925234Z" 2021-09-16T19:44:26.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:26.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:26.350 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-16T19:44:26.350 INFO:teuthology.orchestra.run.smithi118.stdout: "last_update": "2021-09-16T19:44:00.926527Z" 2021-09-16T19:44:26.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:26.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:26.351 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:44:26.351 INFO:teuthology.orchestra.run.smithi118.stdout: "last_update": "0.000000" 2021-09-16T19:44:26.352 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:26.352 INFO:teuthology.orchestra.run.smithi118.stdout:] 2021-09-16T19:44:26.353 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-09-16T19:44:26.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.355 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:26.400 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:26.401 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:26.460 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:44:26.460 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:44:26.461 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:44:26.461 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:44:26.461 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:44:26.461 INFO:teuthology.orchestra.run.smithi109.stdout: "instance_id": 2454120434684624778 2021-09-16T19:44:26.462 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.462 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:44:26.462 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.462 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:44:26.463 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:26.463 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "incremental-sync", 2021-09-16T19:44:26.463 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-16T19:44:26.463 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:26.463 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:26.464 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:26.464 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:43:38.922787Z" 2021-09-16T19:44:26.464 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.464 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.465 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.465 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:44:26.465 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:26.465 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "incremental-sync", 2021-09-16T19:44:26.466 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-16T19:44:26.466 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:26.466 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:26.466 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:26.467 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:44:00.925234Z" 2021-09-16T19:44:26.467 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.467 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.467 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.468 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:44:26.468 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:26.468 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "incremental-sync", 2021-09-16T19:44:26.468 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-16T19:44:26.468 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:26.469 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:26.469 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:26.469 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:44:00.926527Z" 2021-09-16T19:44:26.469 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.470 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.470 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.470 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:44:26.470 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:26.470 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "incremental-sync", 2021-09-16T19:44:26.471 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:44:26.471 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:26.471 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:26.471 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:26.472 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000" 2021-09-16T19:44:26.472 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.472 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.472 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:44:26.472 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.473 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:44:26.473 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 0, 2021-09-16T19:44:26.473 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 0 2021-09-16T19:44:26.473 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.474 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:44:26.475 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-09-16T19:44:26.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.479 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:26.521 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:26.522 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:26.574 INFO:teuthology.orchestra.run.smithi118.stdout:[ 2021-09-16T19:44:26.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:26.575 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-16T19:44:26.575 INFO:teuthology.orchestra.run.smithi118.stdout: "last_update": "2021-09-16T19:43:38.922787Z" 2021-09-16T19:44:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-16T19:44:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: "last_update": "2021-09-16T19:44:00.925234Z" 2021-09-16T19:44:26.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-16T19:44:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: "last_update": "2021-09-16T19:44:00.926527Z" 2021-09-16T19:44:26.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:44:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: "last_update": "0.000000" 2021-09-16T19:44:26.578 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:26.578 INFO:teuthology.orchestra.run.smithi118.stdout:] 2021-09-16T19:44:26.580 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-09-16T19:44:26.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.581 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:26.625 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:26.625 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:26.681 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:44:26.681 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:44:26.682 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:44:26.682 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:44:26.682 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:44:26.682 INFO:teuthology.orchestra.run.smithi109.stdout: "instance_id": 11322070522861530273 2021-09-16T19:44:26.683 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.683 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:44:26.683 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.683 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:44:26.684 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:26.684 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "incremental-sync", 2021-09-16T19:44:26.684 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-16T19:44:26.684 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:26.685 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:26.685 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:26.685 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:43:38.922787Z" 2021-09-16T19:44:26.685 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.686 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.686 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.686 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:44:26.686 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:26.687 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "incremental-sync", 2021-09-16T19:44:26.687 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-16T19:44:26.687 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:26.688 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:26.688 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:26.688 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:44:00.925234Z" 2021-09-16T19:44:26.688 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.688 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.689 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.689 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:44:26.689 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:26.689 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "incremental-sync", 2021-09-16T19:44:26.690 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-16T19:44:26.690 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:26.690 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:26.690 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:26.691 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:44:00.926527Z" 2021-09-16T19:44:26.691 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.691 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.691 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.691 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:44:26.692 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:26.692 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "incremental-sync", 2021-09-16T19:44:26.692 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:44:26.692 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:26.693 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:26.693 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:26.693 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000" 2021-09-16T19:44:26.693 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.693 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.694 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:44:26.694 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.694 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:44:26.694 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 0, 2021-09-16T19:44:26.695 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 0 2021-09-16T19:44:26.695 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.695 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:44:26.696 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-09-16T19:44:26.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.698 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:26.739 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:26.739 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:26.813 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:44:26.814 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.814 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-16T19:44:26.814 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:43:38.880372Z" 2021-09-16T19:44:26.814 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.815 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.815 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-16T19:44:26.815 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:44:00.895136Z" 2021-09-16T19:44:26.815 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.816 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.816 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-16T19:44:26.816 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:44:00.896546Z" 2021-09-16T19:44:26.816 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.816 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.817 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:44:26.817 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:44:26.817 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.817 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:44:26.818 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-09-16T19:44:26.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.820 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:26.903 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:26.903 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:26.953 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:44:26.953 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:44:26.954 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:44:26.954 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:44:26.954 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:44:26.955 INFO:teuthology.orchestra.run.smithi109.stdout: "instance_id": 1350340676335014153 2021-09-16T19:44:26.955 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.956 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:44:26.956 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.956 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:44:26.956 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:26.957 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "incremental-sync", 2021-09-16T19:44:26.957 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-16T19:44:26.957 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:26.958 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:26.958 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:26.958 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:43:38.880372Z" 2021-09-16T19:44:26.958 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.959 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.959 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.959 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:44:26.959 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:26.960 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "incremental-sync", 2021-09-16T19:44:26.960 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-16T19:44:26.960 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:26.960 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:26.960 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:26.961 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:44:00.895136Z" 2021-09-16T19:44:26.961 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.961 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.961 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.962 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:44:26.962 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:26.962 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "incremental-sync", 2021-09-16T19:44:26.962 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-16T19:44:26.962 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:26.963 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:26.963 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:26.963 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:44:00.896546Z" 2021-09-16T19:44:26.963 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.964 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.964 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:26.964 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:44:26.964 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:26.964 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "incremental-sync", 2021-09-16T19:44:26.965 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:44:26.965 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:26.965 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:26.965 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:26.966 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000" 2021-09-16T19:44:26.966 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.966 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.966 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:44:26.967 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:26.967 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:44:26.967 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 0, 2021-09-16T19:44:26.967 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 0 2021-09-16T19:44:26.967 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:26.968 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:44:26.969 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-09-16T19:44:26.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:26.971 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:27.051 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:27.052 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:27.130 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:44:27.131 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:27.131 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-16T19:44:27.131 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:43:38.880372Z" 2021-09-16T19:44:27.131 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:27.132 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:27.132 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-16T19:44:27.132 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:44:00.895136Z" 2021-09-16T19:44:27.132 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:27.133 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:27.133 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-16T19:44:27.133 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:44:00.896546Z" 2021-09-16T19:44:27.133 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:27.134 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:27.134 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:44:27.134 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:44:27.135 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:27.135 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:44:27.136 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-09-16T19:44:27.136 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:27.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:27.138 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:27.184 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:27.184 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:27.234 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:44:27.235 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:44:27.235 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:44:27.235 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:44:27.235 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:44:27.236 INFO:teuthology.orchestra.run.smithi118.stdout: "instance_id": 15099685286183975446 2021-09-16T19:44:27.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:27.236 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:44:27.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:27.236 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:44:27.237 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:27.237 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "incremental-sync", 2021-09-16T19:44:27.237 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-16T19:44:27.237 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:27.238 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:27.238 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:27.238 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:43:38.880372Z" 2021-09-16T19:44:27.238 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:27.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:27.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:27.239 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:44:27.239 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:27.240 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "incremental-sync", 2021-09-16T19:44:27.240 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-16T19:44:27.240 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:27.240 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:27.240 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:27.241 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:00.895136Z" 2021-09-16T19:44:27.241 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:27.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:27.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:27.242 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:44:27.242 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:27.242 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "incremental-sync", 2021-09-16T19:44:27.242 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-16T19:44:27.242 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:27.243 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:27.243 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:27.243 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:00.896546Z" 2021-09-16T19:44:27.243 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:27.244 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:27.244 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:27.244 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:44:27.244 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:27.245 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "incremental-sync", 2021-09-16T19:44:27.245 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:44:27.245 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:27.245 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:27.245 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:27.246 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000" 2021-09-16T19:44:27.246 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:27.246 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:27.246 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:44:27.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:27.247 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:44:27.247 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 0, 2021-09-16T19:44:27.247 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 0 2021-09-16T19:44:27.247 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:27.248 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:44:27.249 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-09-16T19:44:27.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2021-09-16T19:44:27.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:27.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:27.271 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:27.312 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:27.312 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:27.365 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:44:27.378 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:44:27.378 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:27.379 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:44:27.379 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:44:27.380 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:27.380 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:27.380 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:44:27.381 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:44:27.381 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:27.382 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:27.382 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821467.265880_133.1", 2021-09-16T19:44:27.383 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:44:27.265880Z" 2021-09-16T19:44:27.383 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:27.383 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:27.384 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:44:27.385 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:17.633408Z" 2021-09-16T19:44:27.385 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:27.385 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:44:27.388 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:44:27.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:27.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:27.390 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:27.434 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:27.434 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:27.481 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:44:27.482 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:44:27.482 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:44:27.482 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:44:27.483 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:44:27.483 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:44:27.483 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:27.483 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:27.484 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:44:27.484 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:27.484 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:44:27.484 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:27.485 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:27.485 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:44:27.485 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:27.486 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:44:27.486 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:27.486 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:44:27.486 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:27.486 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:27.487 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:27.487 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:27.487 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:44:27.487 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:27.488 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:27.488 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:44:27.488 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:27.488 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:27.489 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:27.489 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:44:27.489 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:27.489 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:27.489 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:27.490 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:27.490 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:44:27.490 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:27.490 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:27.491 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821467.258761_130.1", 2021-09-16T19:44:27.491 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:27.491 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:27.491 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:27.491 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:27.258761Z", 2021-09-16T19:44:27.492 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:27.492 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:27.492 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:27.493 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:27.493 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:44:27.494 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:27.494 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:27.495 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:44:27.495 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:27.495 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:27.495 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:27.495 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:44:27.496 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:27.496 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:27.496 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:27.496 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:44:27.497 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:27.497 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:44:27.497 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:44:27.497 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:44:27.497 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:27.498 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:44:27.500 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821467.265880_133.1 target=1_1631821467.258761_130.1 2021-09-16T19:44:32.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:32.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:32.502 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:32.552 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:32.552 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:32.606 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:44:32.606 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:44:32.606 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:44:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:44:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:44:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:44:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:32.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:44:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:44:32.608 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:44:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:44:32.609 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:44:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:32.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:44:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:44:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:44:32.612 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:44:32.613 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821467.265880_133.1", 2021-09-16T19:44:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:32.614 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:27.265880Z", 2021-09-16T19:44:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:32.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:44:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:32.616 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:44:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:32.617 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:44:32.618 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:32.618 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:32.618 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:32.618 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:44:32.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:44:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:44:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:44:32.619 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:32.619 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:44:32.620 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:44:32.621 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:44:32.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:32.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:32.623 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:32.671 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:32.671 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:32.733 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:44:32.734 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:44:32.734 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:44:32.735 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:44:32.735 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:44:32.736 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:44:32.736 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:44:32.736 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:32.737 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:44:32.737 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:32.738 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:44:32.738 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:32.739 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:44:32.739 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:44:32.740 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:32.740 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:44:32.740 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:32.741 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:44:32.741 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:44:32.742 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:32.742 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:32.743 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:32.743 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:44:32.744 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:32.744 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:44:32.744 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:44:32.745 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:32.745 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:32.745 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:32.746 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:44:32.746 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:44:32.747 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:32.747 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:32.748 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:32.748 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:44:32.749 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:32.749 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:44:32.749 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821467.265880_133.1", 2021-09-16T19:44:32.750 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:32.750 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:32.751 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:32.751 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:44:27.265880Z", 2021-09-16T19:44:32.752 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:44:32.752 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:32.752 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:32.753 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:32.753 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:44:32.754 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:32.754 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:44:32.755 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:44:32.755 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:32.756 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:32.756 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:32.756 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:44:32.757 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:44:32.757 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:32.758 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:32.758 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:44:32.759 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:32.759 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:44:32.760 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:44:32.760 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:44:32.760 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:32.761 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:44:32.763 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-16T19:44:32.764 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-16T19:44:32.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:32.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:32.767 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:32.813 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:32.813 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:32.861 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:44:32.861 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:44:32.861 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:44:32.861 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:44:32.862 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:44:32.862 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:44:32.862 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:32.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:32.863 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:44:32.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:32.863 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:44:32.863 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:32.864 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:32.864 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:44:32.864 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:32.864 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:44:32.864 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:32.865 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:44:32.865 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:32.865 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:32.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:32.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:32.866 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:44:32.866 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:32.866 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:32.866 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:44:32.867 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:32.867 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:32.867 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:32.867 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:44:32.868 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:32.868 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:32.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:32.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:32.868 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:44:32.869 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:32.869 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:32.869 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821467.265880_133.1", 2021-09-16T19:44:32.869 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:32.869 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:32.870 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:32.870 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:27.265880Z", 2021-09-16T19:44:32.870 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:32.870 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:32.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:32.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:32.871 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:44:32.871 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:32.871 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:32.872 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:44:32.872 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:32.872 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:32.872 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:32.873 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:44:32.873 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:32.873 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:32.873 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:32.873 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:44:32.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:32.874 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:44:32.874 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:44:32.874 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:44:32.875 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:32.875 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:44:32.876 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-16T19:44:32.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:32.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:32.924 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:32.974 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:32.975 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:32.992 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:32.991+0000 7fd0a474e380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:32.993 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:32.991+0000 7fd0a474e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:32.993 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:32.991+0000 7fd0a474e380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:44:32.993 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:32.991+0000 7fd0a474e380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:33.016 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:33.015+0000 7fd0a474e380 1 waiting for incremental sync to catch up: 2021-09-16T19:44:33.017 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , , , ] 2021-09-16T19:44:33.017 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000001.16.5, , ] 2021-09-16T19:44:38.018 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:38.016+0000 7fd0a474e380 1 bucket sync caught up with source: 2021-09-16T19:44:38.018 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000001.16.5, , ] 2021-09-16T19:44:38.019 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000001.16.5, , ] 2021-09-16T19:44:38.019 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:38.016+0000 7fd0a474e380 0 bucket checkpoint complete 2021-09-16T19:44:38.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:38.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:38.036 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:38.090 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:38.090 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:38.108 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.107+0000 7ff660515380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:38.108 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.107+0000 7ff660515380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:38.109 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.107+0000 7ff660515380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:44:38.109 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.107+0000 7ff660515380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:38.148 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.146+0000 7ff660515380 1 bucket sync caught up with source: 2021-09-16T19:44:38.148 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000001.16.5, , ] 2021-09-16T19:44:38.148 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000001.16.5, , ] 2021-09-16T19:44:38.148 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.146+0000 7ff660515380 0 bucket checkpoint complete 2021-09-16T19:44:38.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:38.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:38.165 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:38.219 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:38.219 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:38.239 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:38.238+0000 7f61fe3db380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:38.239 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:38.238+0000 7f61fe3db380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:38.240 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:38.238+0000 7f61fe3db380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:44:38.240 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:38.238+0000 7f61fe3db380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:38.264 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:38.263+0000 7f61fe3db380 1 bucket sync caught up with source: 2021-09-16T19:44:38.265 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000001.16.5, , ] 2021-09-16T19:44:38.265 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000001.16.5, , ] 2021-09-16T19:44:38.265 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:38.263+0000 7f61fe3db380 0 bucket checkpoint complete 2021-09-16T19:44:38.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:38.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:38.278 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:38.328 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:38.328 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:38.346 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.344+0000 7fce72ff0380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:38.346 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.344+0000 7fce72ff0380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:38.347 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.344+0000 7fce72ff0380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:44:38.347 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.344+0000 7fce72ff0380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:38.380 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.378+0000 7fce72ff0380 1 bucket sync caught up with source: 2021-09-16T19:44:38.380 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000001.181.5, , ] 2021-09-16T19:44:38.380 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000001.181.5, , ] 2021-09-16T19:44:38.380 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.378+0000 7fce72ff0380 0 bucket checkpoint complete 2021-09-16T19:44:38.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:38.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:38.399 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:38.447 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:38.447 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:38.464 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.463+0000 7f8846621380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:38.464 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.463+0000 7f8846621380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:38.464 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.463+0000 7f8846621380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:44:38.465 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.463+0000 7f8846621380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:38.508 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.507+0000 7f8846621380 1 bucket sync caught up with source: 2021-09-16T19:44:38.508 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000001.181.5, , ] 2021-09-16T19:44:38.508 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000001.181.5, , ] 2021-09-16T19:44:38.509 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.507+0000 7f8846621380 0 bucket checkpoint complete 2021-09-16T19:44:38.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:38.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:38.532 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:38.579 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:38.580 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:38.597 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.595+0000 7f74ec645380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:38.597 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.595+0000 7f74ec645380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:38.597 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.595+0000 7f74ec645380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:44:38.597 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.595+0000 7f74ec645380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:38.624 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.623+0000 7f74ec645380 1 bucket sync caught up with source: 2021-09-16T19:44:38.624 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000001.189.5, , ] 2021-09-16T19:44:38.624 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000001.189.5, , ] 2021-09-16T19:44:38.625 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:38.623+0000 7f74ec645380 0 bucket checkpoint complete 2021-09-16T19:44:38.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:38.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:38.640 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:38.693 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:38.693 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:38.712 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:38.711+0000 7ffb6d2c2380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:38.712 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:38.711+0000 7ffb6d2c2380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:38.712 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:38.711+0000 7ffb6d2c2380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:44:38.712 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:38.711+0000 7ffb6d2c2380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:38.737 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:38.737+0000 7ffb6d2c2380 1 bucket sync caught up with source: 2021-09-16T19:44:38.738 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000001.189.5, , ] 2021-09-16T19:44:38.738 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000001.189.5, , ] 2021-09-16T19:44:38.738 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:38.737+0000 7ffb6d2c2380 0 bucket checkpoint complete 2021-09-16T19:44:38.750 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-20 zones={test-zone2, test-zone1} 2021-09-16T19:44:38.755 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:44:38.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:38.803 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:44:38.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:38.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:38.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:38.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:38.973 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-20 zones={test-zone2, test-zone1} 2021-09-16T19:44:38.974 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-20 zones={test-zone3, test-zone1} 2021-09-16T19:44:38.979 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:44:39.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:39.024 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:44:39.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:39.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:39.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:39.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:39.197 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-20 zones={test-zone3, test-zone1} 2021-09-16T19:44:39.198 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-20 zones={test-zone3, test-zone2} 2021-09-16T19:44:39.203 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:44:39.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:39.248 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:44:39.250 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:39.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:39.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:39.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:39.458 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-20 zones={test-zone3, test-zone2} 2021-09-16T19:44:39.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:39.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:39.464 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:39.505 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:39.506 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:39.547 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:44:39.559 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:44:39.560 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:39.560 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:44:39.560 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:44:39.561 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:39.561 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:39.561 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:44:39.561 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:44:39.561 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:39.562 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:39.562 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821479.462610_135.1", 2021-09-16T19:44:39.562 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:44:39.462610Z" 2021-09-16T19:44:39.562 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:39.563 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:39.563 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:44:39.563 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:17.633408Z" 2021-09-16T19:44:39.563 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:39.564 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:44:39.565 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:44:39.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:39.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:39.567 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:39.611 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:39.612 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:39.659 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:44:39.660 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:44:39.660 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:44:39.660 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:44:39.661 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:44:39.661 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:44:39.661 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:39.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:39.661 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:44:39.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:39.662 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:44:39.662 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:39.662 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:39.663 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:44:39.663 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:39.663 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:44:39.663 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:39.664 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:44:39.664 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:39.664 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:39.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:39.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:39.665 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:44:39.665 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:39.665 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:39.665 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:44:39.666 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:39.666 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:39.666 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:39.666 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:44:39.667 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:39.667 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:39.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:39.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:39.668 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:44:39.668 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:39.668 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:39.668 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821479.460113_134.1", 2021-09-16T19:44:39.668 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:39.669 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:39.669 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:39.669 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:39.460113Z", 2021-09-16T19:44:39.669 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:39.670 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:39.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:39.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:39.670 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:44:39.670 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:39.671 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:39.671 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:44:39.671 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:39.671 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:39.672 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:39.672 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:44:39.672 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:39.672 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:39.672 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:39.673 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:44:39.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:39.673 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:44:39.673 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:44:39.673 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:44:39.674 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:39.674 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:44:39.675 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821479.462610_135.1 target=1_1631821479.460113_134.1 2021-09-16T19:44:44.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:44.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:44.678 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:44.727 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:44.728 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:44.786 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:44:44.786 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:44:44.786 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:44:44.787 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:44:44.787 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:44:44.787 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:44:44.787 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:44.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:44.788 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:44:44.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:44.788 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:44:44.788 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:44.789 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:44.789 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:44:44.789 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:44.789 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:44:44.790 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:44.790 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:44:44.790 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:44.790 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:44.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:44.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:44.791 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:44:44.791 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:44.791 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:44.792 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:44:44.792 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:44.792 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:44.792 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:44.792 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:44:44.793 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:44.793 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:44.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:44.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:44.793 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:44:44.794 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:44.794 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:44.794 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821479.462610_135.1", 2021-09-16T19:44:44.794 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:44.795 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:44.795 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:44.795 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:39.462610Z", 2021-09-16T19:44:44.795 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:44.795 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:44.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:44.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:44.796 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:44:44.796 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:44.796 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:44.797 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:44:44.797 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:44.798 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:44.798 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:44.798 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:44:44.798 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:44.798 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:44.799 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:44.799 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:44:44.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:44.799 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:44:44.800 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:44:44.800 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:44:44.800 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:44.800 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:44:44.801 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:44:44.802 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:44:44.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:44.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:44.803 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:44.853 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:44.854 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:44.918 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:44:44.919 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:44:44.919 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:44:44.920 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:44:44.920 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:44:44.920 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:44:44.921 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:44:44.921 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:44.921 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:44:44.921 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:44.922 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:44:44.922 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:44.923 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:44:44.923 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:44:44.924 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:44.924 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:44:44.925 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:44.925 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:44:44.925 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:44:44.926 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:44.926 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:44.927 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:44.927 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:44:44.928 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:44.928 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:44:44.928 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:44:44.929 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:44.930 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:44.930 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:44.931 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:44:44.931 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:44:44.932 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:44.932 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:44.933 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:44.933 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:44:44.933 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:44.933 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:44:44.934 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821479.462610_135.1", 2021-09-16T19:44:44.934 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:44.934 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:44.935 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:44.935 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:44:39.462610Z", 2021-09-16T19:44:44.935 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:44:44.935 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:44.935 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:44.936 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:44.936 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:44:44.936 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:44.937 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:44:44.937 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:44:44.937 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:44.938 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:44.938 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:44.938 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:44:44.938 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:44:44.939 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:44.939 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:44.939 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:44:44.939 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:44.940 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:44:44.940 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:44:44.940 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:44:44.940 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:44.941 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:44:44.943 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-16T19:44:44.943 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-16T19:44:44.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:44.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:44.947 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:44.994 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:44.994 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:45.045 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:44:45.045 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:44:45.045 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:44:45.046 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:44:45.046 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:44:45.047 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:44:45.047 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:45.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:45.048 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:44:45.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:45.048 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:44:45.048 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:45.048 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:45.049 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:44:45.049 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:45.049 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:44:45.049 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:45.050 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:44:45.050 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:45.050 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:45.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:45.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:45.051 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:44:45.051 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:45.051 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:45.051 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:44:45.052 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:45.052 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:45.053 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:45.053 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:44:45.053 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:45.054 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:45.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:45.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:45.054 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:44:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821479.462610_135.1", 2021-09-16T19:44:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:39.462610Z", 2021-09-16T19:44:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:45.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:45.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:45.057 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:44:45.057 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:45.057 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:45.058 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:44:45.058 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:45.058 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:45.058 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:45.059 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:44:45.059 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:45.059 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:45.059 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:45.059 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:44:45.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:45.060 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:44:45.060 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:44:45.060 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:44:45.061 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:45.061 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:44:45.062 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-16T19:44:45.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:45.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:45.117 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:45.168 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:45.168 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:45.185 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:45.185+0000 7f55e8dbc380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:45.186 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:45.185+0000 7f55e8dbc380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:45.186 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:45.185+0000 7f55e8dbc380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:44:45.186 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:45.185+0000 7f55e8dbc380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:45.210 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:45.210+0000 7f55e8dbc380 1 waiting for incremental sync to catch up: 2021-09-16T19:44:45.211 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000001.16.5, , ] 2021-09-16T19:44:45.211 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000003.19.12, , ] 2021-09-16T19:44:50.212 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:50.210+0000 7f55e8dbc380 1 bucket sync caught up with source: 2021-09-16T19:44:50.212 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000003.19.12, , ] 2021-09-16T19:44:50.212 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000003.19.12, , ] 2021-09-16T19:44:50.213 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:50.210+0000 7f55e8dbc380 0 bucket checkpoint complete 2021-09-16T19:44:50.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:50.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:50.230 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:50.286 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:50.287 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:50.307 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.305+0000 7f5e45f66380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:50.307 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.305+0000 7f5e45f66380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:50.307 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.305+0000 7f5e45f66380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:44:50.307 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.305+0000 7f5e45f66380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:50.351 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.350+0000 7f5e45f66380 1 bucket sync caught up with source: 2021-09-16T19:44:50.351 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000003.19.12, , ] 2021-09-16T19:44:50.352 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000003.19.12, , ] 2021-09-16T19:44:50.352 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.350+0000 7f5e45f66380 0 bucket checkpoint complete 2021-09-16T19:44:50.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:50.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:50.368 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:50.425 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:50.425 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:50.445 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:50.444+0000 7fec16ff2380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:50.445 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:50.444+0000 7fec16ff2380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:50.446 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:50.444+0000 7fec16ff2380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:44:50.446 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:50.444+0000 7fec16ff2380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:50.471 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:50.470+0000 7fec16ff2380 1 bucket sync caught up with source: 2021-09-16T19:44:50.472 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000003.19.12, , ] 2021-09-16T19:44:50.472 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000003.19.12, , ] 2021-09-16T19:44:50.472 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:50.470+0000 7fec16ff2380 0 bucket checkpoint complete 2021-09-16T19:44:50.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:50.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:50.486 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:50.536 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:50.537 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:50.553 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.552+0000 7f7ca7aa1380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:50.554 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.552+0000 7f7ca7aa1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:50.554 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.552+0000 7f7ca7aa1380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:44:50.554 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.552+0000 7f7ca7aa1380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:50.588 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.586+0000 7f7ca7aa1380 1 bucket sync caught up with source: 2021-09-16T19:44:50.588 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000004.196.9, , ] 2021-09-16T19:44:50.589 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000004.196.9, , ] 2021-09-16T19:44:50.589 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.586+0000 7f7ca7aa1380 0 bucket checkpoint complete 2021-09-16T19:44:50.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:50.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:50.602 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:50.650 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:50.650 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:50.667 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.666+0000 7f6ccd1e3380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:50.668 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.666+0000 7f6ccd1e3380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:50.672 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.666+0000 7f6ccd1e3380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:44:50.672 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.666+0000 7f6ccd1e3380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:50.692 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.691+0000 7f6ccd1e3380 1 bucket sync caught up with source: 2021-09-16T19:44:50.693 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000004.196.9, , ] 2021-09-16T19:44:50.693 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000004.196.9, , ] 2021-09-16T19:44:50.693 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.691+0000 7f6ccd1e3380 0 bucket checkpoint complete 2021-09-16T19:44:50.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:50.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:50.708 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:50.756 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:50.757 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:50.774 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.772+0000 7f4562445380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:50.774 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.772+0000 7f4562445380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:50.774 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.772+0000 7f4562445380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:44:50.775 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.772+0000 7f4562445380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:50.814 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.813+0000 7f4562445380 1 bucket sync caught up with source: 2021-09-16T19:44:50.815 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000004.204.9, , ] 2021-09-16T19:44:50.815 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000004.204.9, , ] 2021-09-16T19:44:50.815 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:50.813+0000 7f4562445380 0 bucket checkpoint complete 2021-09-16T19:44:50.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:50.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:50.829 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:50.881 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:50.882 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:50.899 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:50.898+0000 7efccdd79380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:50.900 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:50.898+0000 7efccdd79380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:50.900 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:50.898+0000 7efccdd79380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:44:50.900 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:50.898+0000 7efccdd79380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:50.924 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:50.923+0000 7efccdd79380 1 bucket sync caught up with source: 2021-09-16T19:44:50.924 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000004.204.9, , ] 2021-09-16T19:44:50.925 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000004.204.9, , ] 2021-09-16T19:44:50.925 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:50.923+0000 7efccdd79380 0 bucket checkpoint complete 2021-09-16T19:44:50.937 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-20 zones={test-zone2, test-zone1} 2021-09-16T19:44:50.942 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:44:50.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:50.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:50.988 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:44:50.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:50.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:51.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.274 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-20 zones={test-zone2, test-zone1} 2021-09-16T19:44:51.274 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-20 zones={test-zone3, test-zone1} 2021-09-16T19:44:51.279 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:44:51.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:51.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:51.324 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:44:51.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:51.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:51.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.602 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-20 zones={test-zone3, test-zone1} 2021-09-16T19:44:51.603 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-20 zones={test-zone3, test-zone2} 2021-09-16T19:44:51.608 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:44:51.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:51.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:51.656 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:44:51.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:51.660 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:51.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:51.946 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-20 zones={test-zone3, test-zone2} 2021-09-16T19:44:51.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:51.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:51.952 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:51.994 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:51.994 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:52.046 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:44:52.060 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:44:52.061 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:52.061 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:44:52.061 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:44:52.062 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:52.062 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:52.062 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:44:52.062 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:44:52.062 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:52.063 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:52.063 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821491.950841_137.1", 2021-09-16T19:44:52.063 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:44:51.950841Z" 2021-09-16T19:44:52.064 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:52.064 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:52.064 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:44:52.064 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:17.633408Z" 2021-09-16T19:44:52.064 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:52.065 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:44:52.066 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:44:52.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:52.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:52.067 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:52.111 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:52.111 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:52.158 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:44:52.159 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:44:52.159 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:44:52.159 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:44:52.160 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:44:52.160 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:44:52.160 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:52.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:52.161 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:44:52.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:52.161 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:44:52.161 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:52.161 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:52.162 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:44:52.162 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:52.162 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:44:52.162 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:52.163 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:44:52.163 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:52.163 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:52.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:52.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:52.164 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:44:52.164 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:52.164 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:52.164 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:44:52.165 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:52.165 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:52.165 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:52.165 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:44:52.166 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:52.166 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:52.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:52.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:52.166 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:44:52.167 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:52.167 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:52.167 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821491.948020_136.1", 2021-09-16T19:44:52.167 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:52.168 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:52.168 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:52.168 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:51.948020Z", 2021-09-16T19:44:52.168 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:52.169 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:52.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:52.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:52.169 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:44:52.169 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:52.170 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:52.170 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:44:52.170 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:52.170 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:52.171 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:52.171 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:44:52.171 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:52.171 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:52.172 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:52.172 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:44:52.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:52.172 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:44:52.172 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:44:52.173 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:44:52.173 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:52.173 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:44:52.175 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821491.950841_137.1 target=1_1631821491.948020_136.1 2021-09-16T19:44:57.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:57.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:57.177 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:57.227 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:57.227 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:57.280 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:44:57.281 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:44:57.281 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:44:57.281 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:44:57.282 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:44:57.282 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:44:57.282 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:57.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:57.283 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:44:57.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:57.283 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:44:57.283 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:57.284 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:57.284 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:44:57.284 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:57.284 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:44:57.284 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:57.285 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:44:57.285 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:57.285 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:57.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:57.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:57.286 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:44:57.286 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:57.286 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:57.287 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:44:57.287 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:57.287 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:57.287 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:57.287 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:44:57.288 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:57.288 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:57.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:57.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:57.289 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:44:57.289 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:57.289 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:57.289 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821491.950841_137.1", 2021-09-16T19:44:57.289 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:57.290 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:57.290 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:57.290 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:51.950841Z", 2021-09-16T19:44:57.290 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:57.290 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:57.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:57.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:57.291 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:44:57.291 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:57.292 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:57.292 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:44:57.292 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:57.292 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:57.292 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:57.293 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:44:57.293 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:57.293 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:57.293 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:57.294 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:44:57.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:57.294 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:44:57.294 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:44:57.295 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:44:57.295 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:57.295 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:44:57.296 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:44:57.297 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:44:57.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:57.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:57.298 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:57.347 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:57.347 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:57.411 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:44:57.411 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:44:57.411 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:44:57.411 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:44:57.412 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:44:57.412 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:44:57.412 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:44:57.412 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:57.413 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:44:57.413 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:57.413 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:44:57.413 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:57.413 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:44:57.414 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:44:57.414 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:57.414 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:44:57.414 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:57.415 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:44:57.415 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:44:57.415 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:57.415 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:57.415 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:57.416 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:44:57.416 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:57.416 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:44:57.416 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:44:57.416 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:57.417 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:57.417 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:57.417 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:44:57.417 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:44:57.417 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:57.418 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:57.418 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:57.418 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:44:57.418 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:57.419 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:44:57.419 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821491.950841_137.1", 2021-09-16T19:44:57.419 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:57.419 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:57.420 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:57.420 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:44:51.950841Z", 2021-09-16T19:44:57.420 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:44:57.420 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:57.420 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:57.421 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:57.421 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:44:57.421 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:44:57.421 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:44:57.422 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:44:57.422 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:44:57.422 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:44:57.422 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:44:57.422 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:44:57.423 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:44:57.423 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:57.423 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:57.423 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:44:57.424 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:57.424 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:44:57.424 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:44:57.424 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:44:57.425 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:57.425 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:44:57.426 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-16T19:44:57.426 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-16T19:44:57.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:57.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:57.428 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:57.471 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:57.472 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:57.518 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:44:57.518 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:44:57.519 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:44:57.519 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:44:57.519 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:44:57.519 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:44:57.520 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:57.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:57.520 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:44:57.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:57.521 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:44:57.521 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:57.521 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:57.521 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:44:57.521 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:57.522 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:44:57.522 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:57.522 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:44:57.522 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:57.522 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:57.523 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:57.523 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:57.523 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:44:57.523 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:57.524 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:57.524 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:44:57.524 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:57.524 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:57.524 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:57.525 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:44:57.525 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:57.525 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:57.525 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:57.526 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:57.526 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:44:57.526 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:57.526 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:57.527 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821491.950841_137.1", 2021-09-16T19:44:57.527 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:57.527 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:57.527 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:57.527 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:51.950841Z", 2021-09-16T19:44:57.528 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:57.528 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:57.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:57.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:57.528 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:44:57.529 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:57.529 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:57.529 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:44:57.529 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:57.529 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:57.530 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:57.530 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:44:57.530 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:57.530 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:57.530 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:57.531 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:44:57.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:57.531 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:44:57.531 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:44:57.532 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:44:57.532 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:57.532 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:44:57.533 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-16T19:44:57.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:57.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:57.580 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:57.630 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:57.631 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:57.649 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:57.648+0000 7f7b28ad1380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:57.650 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:57.648+0000 7f7b28ad1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:57.650 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:57.648+0000 7f7b28ad1380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:44:57.650 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:57.648+0000 7f7b28ad1380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:57.674 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:57.673+0000 7f7b28ad1380 1 bucket sync caught up with source: 2021-09-16T19:44:57.675 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000005.23.17, , ] 2021-09-16T19:44:57.675 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000005.23.17, , ] 2021-09-16T19:44:57.675 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:57.673+0000 7f7b28ad1380 0 bucket checkpoint complete 2021-09-16T19:44:57.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:57.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:57.688 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:57.739 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:57.739 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:57.757 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:57.756+0000 7f170604c380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:57.758 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:57.756+0000 7f170604c380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:57.758 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:57.756+0000 7f170604c380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:44:57.758 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:57.756+0000 7f170604c380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:57.799 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:57.798+0000 7f170604c380 1 bucket sync caught up with source: 2021-09-16T19:44:57.799 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000005.23.17, , ] 2021-09-16T19:44:57.800 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000005.23.17, , ] 2021-09-16T19:44:57.800 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:57.798+0000 7f170604c380 0 bucket checkpoint complete 2021-09-16T19:44:57.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:57.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:57.814 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:57.863 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:57.863 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:57.880 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:57.879+0000 7f11d5ab6380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:57.881 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:57.879+0000 7f11d5ab6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:57.881 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:57.879+0000 7f11d5ab6380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:44:57.881 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:57.879+0000 7f11d5ab6380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:57.904 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:57.903+0000 7f11d5ab6380 1 bucket sync caught up with source: 2021-09-16T19:44:57.905 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000005.23.17, , ] 2021-09-16T19:44:57.905 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000005.23.17, , ] 2021-09-16T19:44:57.905 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:57.903+0000 7f11d5ab6380 0 bucket checkpoint complete 2021-09-16T19:44:57.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:57.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:57.917 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:57.967 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:57.967 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:57.985 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:57.983+0000 7fbf0ec04380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:57.985 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:57.983+0000 7fbf0ec04380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:57.985 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:57.983+0000 7fbf0ec04380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:44:57.986 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:57.983+0000 7fbf0ec04380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:58.015 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:58.013+0000 7fbf0ec04380 1 bucket sync caught up with source: 2021-09-16T19:44:58.015 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000007.212.14, , ] 2021-09-16T19:44:58.015 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000007.212.14, , ] 2021-09-16T19:44:58.016 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:58.013+0000 7fbf0ec04380 0 bucket checkpoint complete 2021-09-16T19:44:58.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:58.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:58.035 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:58.084 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:58.084 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:58.102 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:58.100+0000 7f594d97e380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:58.102 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:58.100+0000 7f594d97e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:58.102 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:58.100+0000 7f594d97e380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:44:58.102 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:58.100+0000 7f594d97e380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:58.142 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:58.140+0000 7f594d97e380 1 bucket sync caught up with source: 2021-09-16T19:44:58.142 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000007.212.14, , ] 2021-09-16T19:44:58.142 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000007.212.14, , ] 2021-09-16T19:44:58.142 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:58.140+0000 7f594d97e380 0 bucket checkpoint complete 2021-09-16T19:44:58.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:58.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:58.159 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:58.207 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:58.207 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:58.225 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:58.223+0000 7fa21439d380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:58.225 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:58.223+0000 7fa21439d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:58.226 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:58.223+0000 7fa21439d380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:44:58.226 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:58.223+0000 7fa21439d380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:58.250 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:58.249+0000 7fa21439d380 1 bucket sync caught up with source: 2021-09-16T19:44:58.250 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000007.220.14, , ] 2021-09-16T19:44:58.251 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000007.220.14, , ] 2021-09-16T19:44:58.251 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:44:58.249+0000 7fa21439d380 0 bucket checkpoint complete 2021-09-16T19:44:58.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:58.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:44:58.264 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:44:58.313 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:58.314 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:58.331 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:58.329+0000 7f64a0fcf380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:44:58.331 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:58.329+0000 7f64a0fcf380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:44:58.331 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:58.329+0000 7f64a0fcf380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:44:58.332 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:58.329+0000 7f64a0fcf380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:44:58.355 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:58.354+0000 7f64a0fcf380 1 bucket sync caught up with source: 2021-09-16T19:44:58.356 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000007.220.14, , ] 2021-09-16T19:44:58.356 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000007.220.14, , ] 2021-09-16T19:44:58.356 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:44:58.354+0000 7f64a0fcf380 0 bucket checkpoint complete 2021-09-16T19:44:58.369 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-20 zones={test-zone2, test-zone1} 2021-09-16T19:44:58.374 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:44:58.419 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:58.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:58.420 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:44:58.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:58.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:58.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:58.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:58.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:58.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:58.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:58.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:58.694 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-20 zones={test-zone2, test-zone1} 2021-09-16T19:44:58.694 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-20 zones={test-zone3, test-zone1} 2021-09-16T19:44:58.699 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:44:58.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:58.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:58.748 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:44:58.751 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:58.752 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:58.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:58.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:58.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:58.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:58.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:58.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:59.037 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-20 zones={test-zone3, test-zone1} 2021-09-16T19:44:59.038 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-20 zones={test-zone3, test-zone2} 2021-09-16T19:44:59.043 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:44:59.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:59.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:59.088 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:44:59.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:59.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:44:59.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:59.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:59.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:59.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:59.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:59.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:44:59.373 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-20 zones={test-zone3, test-zone2} 2021-09-16T19:44:59.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2021-09-16T19:44:59.375 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-21 2021-09-16T19:44:59.393 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=htrfwo-22 2021-09-16T19:44:59.422 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=htrfwo-23 2021-09-16T19:44:59.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:59.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:59.483 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:59.524 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:59.525 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:59.570 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:44:59.583 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:44:59.584 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:59.584 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:44:59.584 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:44:59.584 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:59.585 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:59.585 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:44:59.585 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:44:59.585 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:59.586 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:59.586 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821499.475696_155.1", 2021-09-16T19:44:59.586 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:44:59.475696Z" 2021-09-16T19:44:59.586 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:44:59.586 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:44:59.587 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:44:59.587 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:17.633408Z" 2021-09-16T19:44:59.587 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:44:59.587 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:44:59.588 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:44:59.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:59.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:44:59.590 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:44:59.635 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:44:59.636 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:44:59.686 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:44:59.686 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:44:59.687 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:44:59.687 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:44:59.687 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:44:59.687 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:44:59.687 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:59.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:59.688 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:44:59.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:59.688 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:44:59.689 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:59.689 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:59.689 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:44:59.689 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:59.689 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:44:59.690 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:59.690 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:44:59.690 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:59.690 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:59.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:59.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:59.691 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:44:59.691 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:59.691 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:59.692 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:44:59.692 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:59.692 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:59.692 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:59.693 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:44:59.693 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:59.693 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:59.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:59.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:59.694 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:44:59.694 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:59.694 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:59.694 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821499.385786_138.1", 2021-09-16T19:44:59.695 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:59.695 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:59.695 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:59.695 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:59.385786Z", 2021-09-16T19:44:59.695 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:59.696 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:59.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:59.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:44:59.696 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:44:59.696 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:44:59.697 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:44:59.697 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:44:59.697 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:44:59.697 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:44:59.698 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:44:59.698 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:44:59.698 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:44:59.698 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:59.698 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:59.699 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:44:59.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:44:59.699 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:44:59.699 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:44:59.699 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:44:59.700 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:44:59.700 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:44:59.701 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821499.475696_155.1 target=1_1631821499.385786_138.1 2021-09-16T19:45:04.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:04.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:04.704 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:45:04.752 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:04.753 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:04.805 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:45:04.806 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:45:04.806 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:45:04.806 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:45:04.807 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:45:04.807 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:45:04.807 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:04.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:04.808 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:45:04.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:04.808 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:45:04.809 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:04.809 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:04.809 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:45:04.810 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:04.810 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:45:04.810 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:04.811 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:45:04.811 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:04.812 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:04.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:04.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:04.813 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:45:04.813 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:04.813 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:04.814 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:45:04.814 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:04.814 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:04.815 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:04.815 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:45:04.815 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:04.816 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:04.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:04.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:04.816 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:45:04.816 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:04.817 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:04.817 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821499.475696_155.1", 2021-09-16T19:45:04.817 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:04.817 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:04.818 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:04.818 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:59.475696Z", 2021-09-16T19:45:04.818 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:04.818 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:04.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:04.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:04.819 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:45:04.819 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:04.819 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:04.820 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:45:04.820 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:04.820 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:04.820 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:04.821 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:45:04.821 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:04.821 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:04.821 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:04.822 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:45:04.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:04.822 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:45:04.822 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:45:04.822 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:45:04.823 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:04.823 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:45:04.824 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:45:04.825 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:45:04.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:04.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:04.828 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:45:04.879 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:04.879 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:04.936 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:45:04.936 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:45:04.936 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:45:04.937 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:45:04.938 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:45:04.938 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:45:04.939 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:45:04.939 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:45:04.939 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:45:04.940 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:45:04.940 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:45:04.940 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:45:04.940 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:45:04.941 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:45:04.941 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:45:04.941 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:45:04.941 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:45:04.941 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:45:04.942 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:45:04.942 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:45:04.942 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:45:04.942 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:45:04.943 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:45:04.943 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:45:04.943 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:45:04.943 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:45:04.943 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:45:04.944 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:45:04.944 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:45:04.944 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:45:04.944 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:45:04.945 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:45:04.945 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:45:04.945 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:45:04.945 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:45:04.945 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:45:04.946 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:45:04.946 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821499.475696_155.1", 2021-09-16T19:45:04.946 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:45:04.946 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:45:04.947 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:45:04.947 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:44:59.475696Z", 2021-09-16T19:45:04.947 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:45:04.947 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:45:04.947 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:45:04.948 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:45:04.948 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:45:04.948 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:45:04.948 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:45:04.949 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:45:04.949 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:45:04.950 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:45:04.950 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:45:04.950 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:45:04.950 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:45:04.951 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:45:04.951 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:45:04.951 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:45:04.951 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:45:04.952 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:45:04.952 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:45:04.952 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:45:04.953 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:45:04.953 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:45:04.954 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-16T19:45:04.955 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-16T19:45:04.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:04.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:04.956 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:45:05.002 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:05.002 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:05.048 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:45:05.049 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:45:05.049 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:45:05.049 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:45:05.050 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:45:05.050 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:45:05.050 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:05.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:05.051 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:45:05.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:05.051 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:45:05.051 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:05.052 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:05.052 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:45:05.052 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:05.052 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:45:05.052 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:05.053 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:45:05.053 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:05.053 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:05.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:05.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:05.054 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:45:05.054 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:05.054 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:05.054 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:45:05.055 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:05.055 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:05.055 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:05.055 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:45:05.056 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:05.056 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:05.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:05.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:05.056 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:45:05.057 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:05.057 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:05.057 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821499.475696_155.1", 2021-09-16T19:45:05.057 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:05.058 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:05.058 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:05.058 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:44:59.475696Z", 2021-09-16T19:45:05.058 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:05.058 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:05.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:05.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:05.059 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:45:05.059 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:05.060 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:05.060 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:45:05.060 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:05.060 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:05.060 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:05.061 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:45:05.061 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:05.061 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:05.061 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:05.062 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:45:05.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:05.062 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:45:05.062 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:45:05.062 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:45:05.063 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:05.063 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:45:05.064 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-16T19:45:05.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:05.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:05.238 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:05.290 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:05.290 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:05.308 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:05.307+0000 7f6a00347380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:05.308 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:05.307+0000 7f6a00347380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:05.309 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:05.307+0000 7f6a00347380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:45:05.309 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:05.307+0000 7f6a00347380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:05.334 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:05.333+0000 7f6a00347380 1 bucket sync caught up with source: 2021-09-16T19:45:05.335 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000003.40.11, , ] 2021-09-16T19:45:05.335 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000003.40.11, , ] 2021-09-16T19:45:05.335 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:05.333+0000 7f6a00347380 0 bucket checkpoint complete 2021-09-16T19:45:05.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:05.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:05.350 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:05.401 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:05.401 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:05.426 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.425+0000 7efc3b653380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:05.426 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.425+0000 7efc3b653380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:05.427 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.425+0000 7efc3b653380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:45:05.427 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.425+0000 7efc3b653380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:05.480 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.479+0000 7efc3b653380 1 bucket sync caught up with source: 2021-09-16T19:45:05.481 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000003.40.11, , ] 2021-09-16T19:45:05.481 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000003.40.11, , ] 2021-09-16T19:45:05.481 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.479+0000 7efc3b653380 0 bucket checkpoint complete 2021-09-16T19:45:05.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:05.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:05.496 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:05.548 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:05.548 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:05.566 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:05.564+0000 7f5d4d1d5380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:05.566 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:05.564+0000 7f5d4d1d5380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:05.566 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:05.564+0000 7f5d4d1d5380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:45:05.566 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:05.564+0000 7f5d4d1d5380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:05.589 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:05.587+0000 7f5d4d1d5380 1 bucket sync caught up with source: 2021-09-16T19:45:05.589 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000003.40.11, , ] 2021-09-16T19:45:05.590 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000003.40.11, , ] 2021-09-16T19:45:05.590 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:05.587+0000 7f5d4d1d5380 0 bucket checkpoint complete 2021-09-16T19:45:05.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:05.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:05.605 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:05.656 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:05.657 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:05.674 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.673+0000 7fbc49d5f380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:05.675 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.673+0000 7fbc49d5f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:05.675 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.673+0000 7fbc49d5f380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:45:05.675 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.673+0000 7fbc49d5f380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:05.721 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.719+0000 7fbc49d5f380 1 bucket sync caught up with source: 2021-09-16T19:45:05.722 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000002.234.5, , ] 2021-09-16T19:45:05.722 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000002.234.5, , ] 2021-09-16T19:45:05.723 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.719+0000 7fbc49d5f380 0 bucket checkpoint complete 2021-09-16T19:45:05.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:05.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:05.738 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:05.792 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:05.793 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:05.813 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.812+0000 7f858e543380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:05.814 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.812+0000 7f858e543380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:05.814 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.812+0000 7f858e543380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:45:05.814 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.812+0000 7f858e543380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:05.839 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.838+0000 7f858e543380 1 bucket sync caught up with source: 2021-09-16T19:45:05.839 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000002.234.5, , ] 2021-09-16T19:45:05.840 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000002.234.5, , ] 2021-09-16T19:45:05.840 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.838+0000 7f858e543380 0 bucket checkpoint complete 2021-09-16T19:45:05.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:05.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:05.860 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:05.909 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:05.910 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:05.927 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.926+0000 7fa2966fd380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:05.927 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.926+0000 7fa2966fd380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:05.927 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.926+0000 7fa2966fd380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:45:05.927 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.926+0000 7fa2966fd380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:05.962 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.961+0000 7fa2966fd380 1 bucket sync caught up with source: 2021-09-16T19:45:05.963 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000002.263.5, , ] 2021-09-16T19:45:05.963 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000002.263.5, , ] 2021-09-16T19:45:05.963 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:05.961+0000 7fa2966fd380 0 bucket checkpoint complete 2021-09-16T19:45:05.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:05.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:05.981 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:06.030 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:06.031 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:06.048 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.046+0000 7fdcffea6380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:06.048 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.046+0000 7fdcffea6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:06.049 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.046+0000 7fdcffea6380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:45:06.049 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.046+0000 7fdcffea6380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:06.072 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.071+0000 7fdcffea6380 1 bucket sync caught up with source: 2021-09-16T19:45:06.072 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000002.263.5, , ] 2021-09-16T19:45:06.073 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000002.263.5, , ] 2021-09-16T19:45:06.073 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.071+0000 7fdcffea6380 0 bucket checkpoint complete 2021-09-16T19:45:06.085 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-21 zones={test-zone2, test-zone1} 2021-09-16T19:45:06.090 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:06.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:06.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:06.136 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:06.139 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:06.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:06.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:06.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:06.197 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:06.253 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-21 zones={test-zone2, test-zone1} 2021-09-16T19:45:06.254 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-21 zones={test-zone3, test-zone1} 2021-09-16T19:45:06.259 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:06.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:06.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:06.305 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:06.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:06.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:06.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:06.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:06.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:06.421 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-21 zones={test-zone3, test-zone1} 2021-09-16T19:45:06.421 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-21 zones={test-zone3, test-zone2} 2021-09-16T19:45:06.426 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:06.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:06.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:06.476 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:06.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:06.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:06.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:06.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:06.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:06.593 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-21 zones={test-zone3, test-zone2} 2021-09-16T19:45:06.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:06.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:06.595 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:06.644 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:06.645 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:06.664 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.663+0000 7fdfbdcd1380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:06.664 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.663+0000 7fdfbdcd1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:06.665 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.663+0000 7fdfbdcd1380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:45:06.665 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.663+0000 7fdfbdcd1380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:06.690 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.688+0000 7fdfbdcd1380 1 bucket sync caught up with source: 2021-09-16T19:45:06.690 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000002.17.5, , ] 2021-09-16T19:45:06.690 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000002.17.5, , ] 2021-09-16T19:45:06.691 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.688+0000 7fdfbdcd1380 0 bucket checkpoint complete 2021-09-16T19:45:06.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:06.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:06.704 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:06.753 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:06.754 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:06.771 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:06.770+0000 7f24eab95380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:06.772 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:06.770+0000 7f24eab95380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:06.772 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:06.770+0000 7f24eab95380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:45:06.772 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:06.770+0000 7f24eab95380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:06.810 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:06.809+0000 7f24eab95380 1 bucket sync caught up with source: 2021-09-16T19:45:06.811 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000002.17.5, , ] 2021-09-16T19:45:06.811 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000002.17.5, , ] 2021-09-16T19:45:06.811 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:06.809+0000 7f24eab95380 0 bucket checkpoint complete 2021-09-16T19:45:06.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:06.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:06.832 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:06.882 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:06.882 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:06.899 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.898+0000 7efc74ea3380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:06.900 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.898+0000 7efc74ea3380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:06.900 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.898+0000 7efc74ea3380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:45:06.900 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.898+0000 7efc74ea3380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:06.923 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.921+0000 7efc74ea3380 1 bucket sync caught up with source: 2021-09-16T19:45:06.923 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000002.17.5, , ] 2021-09-16T19:45:06.924 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000002.17.5, , ] 2021-09-16T19:45:06.924 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:06.921+0000 7efc74ea3380 0 bucket checkpoint complete 2021-09-16T19:45:06.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:06.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:06.938 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:06.988 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:06.988 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:07.005 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.003+0000 7fdf13dff380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:07.005 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.003+0000 7fdf13dff380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:07.005 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.003+0000 7fdf13dff380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:45:07.005 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.003+0000 7fdf13dff380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:07.049 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.048+0000 7fdf13dff380 1 bucket sync caught up with source: 2021-09-16T19:45:07.049 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000003.277.11, , ] 2021-09-16T19:45:07.050 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000003.277.11, , ] 2021-09-16T19:45:07.050 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.048+0000 7fdf13dff380 0 bucket checkpoint complete 2021-09-16T19:45:07.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:07.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:07.064 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:07.112 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:07.112 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:07.130 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.128+0000 7f7bf9ff7380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:07.130 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.128+0000 7f7bf9ff7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:07.130 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.128+0000 7f7bf9ff7380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:45:07.130 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.128+0000 7f7bf9ff7380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:07.155 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.154+0000 7f7bf9ff7380 1 bucket sync caught up with source: 2021-09-16T19:45:07.156 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000003.277.11, , ] 2021-09-16T19:45:07.156 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000003.277.11, , ] 2021-09-16T19:45:07.156 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.154+0000 7f7bf9ff7380 0 bucket checkpoint complete 2021-09-16T19:45:07.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:07.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:07.169 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:07.217 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:07.218 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:07.235 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.234+0000 7f82d797e380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:07.236 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.234+0000 7f82d797e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:07.236 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.234+0000 7f82d797e380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:45:07.236 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.234+0000 7f82d797e380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:07.261 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.259+0000 7f82d797e380 1 bucket sync caught up with source: 2021-09-16T19:45:07.261 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000002.301.5, , ] 2021-09-16T19:45:07.262 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000002.301.5, , ] 2021-09-16T19:45:07.262 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:07.259+0000 7f82d797e380 0 bucket checkpoint complete 2021-09-16T19:45:07.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:07.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:07.275 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:07.325 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:07.326 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:07.343 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:07.342+0000 7f4497619380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:07.343 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:07.342+0000 7f4497619380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:07.344 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:07.342+0000 7f4497619380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:45:07.344 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:07.342+0000 7f4497619380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:07.367 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:07.366+0000 7f4497619380 1 bucket sync caught up with source: 2021-09-16T19:45:07.368 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000002.301.5, , ] 2021-09-16T19:45:07.368 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000002.301.5, , ] 2021-09-16T19:45:07.368 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:07.366+0000 7f4497619380 0 bucket checkpoint complete 2021-09-16T19:45:07.380 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-22 zones={test-zone2, test-zone1} 2021-09-16T19:45:07.385 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:07.431 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:07.432 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:07.433 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:07.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:07.437 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:07.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:07.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:07.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:07.557 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-22 zones={test-zone2, test-zone1} 2021-09-16T19:45:07.558 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-22 zones={test-zone3, test-zone1} 2021-09-16T19:45:07.562 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:07.611 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:07.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:07.612 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:07.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:07.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:07.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:07.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:07.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:07.729 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-22 zones={test-zone3, test-zone1} 2021-09-16T19:45:07.729 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-22 zones={test-zone3, test-zone2} 2021-09-16T19:45:07.734 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:07.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:07.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:07.784 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:07.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:07.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:07.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:07.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:07.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:07.901 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-22 zones={test-zone3, test-zone2} 2021-09-16T19:45:07.902 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:07.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:07.903 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:07.954 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:07.955 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:07.972 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:07.970+0000 7fc44a45d380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:07.972 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:07.970+0000 7fc44a45d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:07.972 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:07.970+0000 7fc44a45d380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:45:07.973 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:07.970+0000 7fc44a45d380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:07.996 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:07.995+0000 7fc44a45d380 1 bucket sync caught up with source: 2021-09-16T19:45:07.997 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000002.12.5, , ] 2021-09-16T19:45:07.997 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000002.12.5, , ] 2021-09-16T19:45:07.997 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:07.995+0000 7fc44a45d380 0 bucket checkpoint complete 2021-09-16T19:45:08.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:08.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:08.010 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:08.059 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:08.059 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:08.077 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.075+0000 7f9c39a49380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:08.077 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.075+0000 7f9c39a49380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:08.077 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.075+0000 7f9c39a49380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:45:08.078 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.075+0000 7f9c39a49380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:08.114 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.113+0000 7f9c39a49380 1 bucket sync caught up with source: 2021-09-16T19:45:08.114 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000002.12.5, , ] 2021-09-16T19:45:08.115 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000002.12.5, , ] 2021-09-16T19:45:08.115 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.113+0000 7f9c39a49380 0 bucket checkpoint complete 2021-09-16T19:45:08.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:08.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:08.128 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:08.177 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:08.177 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:08.195 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:08.193+0000 7f67f16ce380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:08.195 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:08.193+0000 7f67f16ce380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:08.195 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:08.193+0000 7f67f16ce380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:45:08.195 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:08.193+0000 7f67f16ce380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:08.217 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:08.216+0000 7f67f16ce380 1 bucket sync caught up with source: 2021-09-16T19:45:08.218 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000002.12.5, , ] 2021-09-16T19:45:08.218 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000002.12.5, , ] 2021-09-16T19:45:08.218 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:08.216+0000 7f67f16ce380 0 bucket checkpoint complete 2021-09-16T19:45:08.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:08.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:08.230 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:08.279 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:08.280 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:08.297 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.295+0000 7f62cdf65380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:08.297 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.295+0000 7f62cdf65380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:08.297 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.295+0000 7f62cdf65380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:45:08.298 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.295+0000 7f62cdf65380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:08.323 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.322+0000 7f62cdf65380 1 bucket sync caught up with source: 2021-09-16T19:45:08.324 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000002.281.5, , ] 2021-09-16T19:45:08.324 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000002.281.5, , ] 2021-09-16T19:45:08.324 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.322+0000 7f62cdf65380 0 bucket checkpoint complete 2021-09-16T19:45:08.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:08.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:08.337 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:08.385 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:08.386 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:08.403 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.402+0000 7f300684a380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:08.403 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.402+0000 7f300684a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:08.404 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.402+0000 7f300684a380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:45:08.404 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.402+0000 7f300684a380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:08.429 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.427+0000 7f300684a380 1 bucket sync caught up with source: 2021-09-16T19:45:08.429 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000002.281.5, , ] 2021-09-16T19:45:08.429 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000002.281.5, , ] 2021-09-16T19:45:08.429 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.427+0000 7f300684a380 0 bucket checkpoint complete 2021-09-16T19:45:08.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:08.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:08.445 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:08.493 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:08.494 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:08.511 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.510+0000 7f3bb1a5d380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:08.511 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.510+0000 7f3bb1a5d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:08.512 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.510+0000 7f3bb1a5d380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:45:08.512 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.510+0000 7f3bb1a5d380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:08.543 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.541+0000 7f3bb1a5d380 1 bucket sync caught up with source: 2021-09-16T19:45:08.543 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000003.297.11, , ] 2021-09-16T19:45:08.543 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000003.297.11, , ] 2021-09-16T19:45:08.543 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:08.541+0000 7f3bb1a5d380 0 bucket checkpoint complete 2021-09-16T19:45:08.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:08.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:08.562 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:08.612 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:08.612 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:08.629 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:08.628+0000 7f4b65b33380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:08.630 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:08.628+0000 7f4b65b33380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:08.630 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:08.628+0000 7f4b65b33380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:45:08.630 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:08.628+0000 7f4b65b33380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:08.653 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:08.652+0000 7f4b65b33380 1 bucket sync caught up with source: 2021-09-16T19:45:08.654 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000003.297.11, , ] 2021-09-16T19:45:08.654 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000003.297.11, , ] 2021-09-16T19:45:08.654 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:08.652+0000 7f4b65b33380 0 bucket checkpoint complete 2021-09-16T19:45:08.667 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-23 zones={test-zone2, test-zone1} 2021-09-16T19:45:08.672 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:08.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:08.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:08.716 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:08.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:08.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:08.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:08.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:08.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:08.829 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-23 zones={test-zone2, test-zone1} 2021-09-16T19:45:08.830 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-23 zones={test-zone3, test-zone1} 2021-09-16T19:45:08.835 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:08.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:08.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:08.880 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:08.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:08.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:08.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:08.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:08.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:09.005 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-23 zones={test-zone3, test-zone1} 2021-09-16T19:45:09.005 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-23 zones={test-zone3, test-zone2} 2021-09-16T19:45:09.010 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:09.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:09.056 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:09.056 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:09.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:09.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:09.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:09.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:09.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:09.187 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-23 zones={test-zone3, test-zone2} 2021-09-16T19:45:09.188 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2021-09-16T19:45:09.189 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-24 2021-09-16T19:45:09.206 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=htrfwo-25 2021-09-16T19:45:09.241 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=htrfwo-26 2021-09-16T19:45:09.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:09.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:09.325 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:45:09.367 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:09.367 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:09.415 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:45:09.427 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:45:09.428 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:45:09.428 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:45:09.429 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:45:09.429 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:45:09.430 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:45:09.430 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:45:09.430 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:45:09.431 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:45:09.431 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:45:09.432 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821509.321904_179.1", 2021-09-16T19:45:09.432 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:45:09.321904Z" 2021-09-16T19:45:09.432 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:45:09.432 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:45:09.433 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:45:09.433 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:17.633408Z" 2021-09-16T19:45:09.433 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:45:09.434 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:45:09.436 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:45:09.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:09.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:09.437 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:45:09.480 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:09.480 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:09.528 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:45:09.528 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:45:09.529 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:45:09.529 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:45:09.529 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:45:09.530 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:45:09.530 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:09.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:09.530 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:45:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:45:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:45:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:45:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:45:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:45:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:45:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:45:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:09.536 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:09.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:09.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:09.536 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:45:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821509.195746_156.1", 2021-09-16T19:45:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:09.195746Z", 2021-09-16T19:45:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:09.539 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:09.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:09.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:09.539 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:45:09.540 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:09.540 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:09.540 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:45:09.540 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:09.541 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:09.541 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:09.541 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:45:09.541 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:45:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:45:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:45:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:45:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:09.544 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:45:09.545 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821509.321904_179.1 target=1_1631821509.195746_156.1 2021-09-16T19:45:14.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:14.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:14.548 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:45:14.597 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:14.597 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:14.657 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:45:14.657 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:45:14.658 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:45:14.658 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:45:14.658 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:45:14.659 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:45:14.659 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:14.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:14.659 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:45:14.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:14.660 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:45:14.660 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:14.660 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:14.660 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:45:14.661 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:14.661 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:45:14.661 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:14.661 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:45:14.661 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:14.662 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:14.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:14.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:14.662 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:45:14.663 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:14.663 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:14.663 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:45:14.663 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:14.664 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:14.664 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:14.664 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:45:14.664 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:14.664 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:14.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:14.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:14.665 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:45:14.665 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:14.666 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:14.666 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821509.321904_179.1", 2021-09-16T19:45:14.666 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:14.666 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:14.666 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:14.667 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:09.321904Z", 2021-09-16T19:45:14.667 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:14.667 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:14.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:14.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:14.668 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:45:14.668 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:14.668 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:14.668 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:45:14.669 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:14.669 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:14.669 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:14.669 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:45:14.669 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:14.670 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:14.670 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:14.670 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:45:14.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:14.671 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:45:14.671 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:45:14.671 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:45:14.671 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:14.672 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:45:14.673 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:45:14.674 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:45:14.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:14.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:14.675 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:45:14.726 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:14.726 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:14.781 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:45:14.782 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:45:14.782 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:45:14.782 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:45:14.783 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:45:14.783 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:45:14.783 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:45:14.783 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:45:14.783 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:45:14.784 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:45:14.784 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:45:14.784 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:45:14.784 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:45:14.785 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:45:14.785 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:45:14.785 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:45:14.785 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:45:14.785 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:45:14.786 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:45:14.786 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:45:14.786 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:45:14.786 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:45:14.787 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:45:14.787 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:45:14.787 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:45:14.787 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:45:14.788 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:45:14.788 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:45:14.788 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:45:14.789 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:45:14.789 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:45:14.789 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:45:14.789 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:45:14.790 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:45:14.790 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:45:14.790 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:45:14.790 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:45:14.791 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821509.321904_179.1", 2021-09-16T19:45:14.791 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:45:14.791 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:45:14.791 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:45:14.792 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:09.321904Z", 2021-09-16T19:45:14.792 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:45:14.792 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:45:14.792 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:45:14.793 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:45:14.793 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:45:14.793 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:45:14.793 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:45:14.794 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:45:14.794 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:45:14.794 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:45:14.795 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:45:14.795 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:45:14.795 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:45:14.795 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:45:14.795 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:45:14.796 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:45:14.796 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:45:14.796 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:45:14.797 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:45:14.797 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:45:14.797 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:45:14.797 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:45:14.799 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-16T19:45:14.799 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-16T19:45:14.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:14.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:14.801 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:45:14.847 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:14.847 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:14.898 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:45:14.898 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:45:14.898 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:45:14.899 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:45:14.899 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:45:14.899 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:45:14.899 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:14.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:14.900 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:45:14.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:14.900 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:45:14.901 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:14.901 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:14.901 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:45:14.901 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:14.901 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:45:14.902 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:14.902 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:45:14.902 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:14.903 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:14.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:14.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:14.903 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:45:14.904 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:14.904 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:14.904 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:45:14.904 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:14.905 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:14.905 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:14.906 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:45:14.906 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:14.907 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:14.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:14.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:14.908 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:45:14.908 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:14.908 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:14.908 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821509.321904_179.1", 2021-09-16T19:45:14.909 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:14.909 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:14.909 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:14.909 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:09.321904Z", 2021-09-16T19:45:14.909 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:14.910 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:14.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:14.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:14.911 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:45:14.911 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:14.911 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:14.911 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:45:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:45:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:45:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:45:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:45:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:45:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:14.914 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:45:14.916 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-16T19:45:15.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:15.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:15.091 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:15.141 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:15.142 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:15.160 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.158+0000 7f9eb013f380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:15.160 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.158+0000 7f9eb013f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:15.160 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.158+0000 7f9eb013f380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:45:15.161 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.158+0000 7f9eb013f380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:15.185 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.183+0000 7f9eb013f380 1 bucket sync caught up with source: 2021-09-16T19:45:15.185 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000003.13.6, , ] 2021-09-16T19:45:15.185 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000003.13.6, , ] 2021-09-16T19:45:15.186 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.183+0000 7f9eb013f380 0 bucket checkpoint complete 2021-09-16T19:45:15.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:15.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:15.199 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:15.250 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:15.251 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:15.270 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.270+0000 7f913240f380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:15.271 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.270+0000 7f913240f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:15.271 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.270+0000 7f913240f380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:45:15.271 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.270+0000 7f913240f380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:15.302 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.302+0000 7f913240f380 1 bucket sync caught up with source: 2021-09-16T19:45:15.303 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000003.13.6, , ] 2021-09-16T19:45:15.303 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000003.13.6, , ] 2021-09-16T19:45:15.303 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.302+0000 7f913240f380 0 bucket checkpoint complete 2021-09-16T19:45:15.326 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:15.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:15.327 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:15.378 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:15.378 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:15.396 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.394+0000 7fd1fb0f7380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:15.396 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.394+0000 7fd1fb0f7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:15.396 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.394+0000 7fd1fb0f7380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:45:15.396 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.394+0000 7fd1fb0f7380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:15.421 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.419+0000 7fd1fb0f7380 1 bucket sync caught up with source: 2021-09-16T19:45:15.422 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000003.13.6, , ] 2021-09-16T19:45:15.422 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000003.13.6, , ] 2021-09-16T19:45:15.422 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.419+0000 7fd1fb0f7380 0 bucket checkpoint complete 2021-09-16T19:45:15.432 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:15.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:15.434 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:15.484 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:15.484 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:15.502 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.501+0000 7f4afbf63380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:15.502 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.501+0000 7f4afbf63380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:15.502 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.501+0000 7f4afbf63380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:45:15.502 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.501+0000 7f4afbf63380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:15.535 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.535+0000 7f4afbf63380 1 bucket sync caught up with source: 2021-09-16T19:45:15.535 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000001.244.12, , ] 2021-09-16T19:45:15.535 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000001.244.12, , ] 2021-09-16T19:45:15.536 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.535+0000 7f4afbf63380 0 bucket checkpoint complete 2021-09-16T19:45:15.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:15.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:15.550 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:15.602 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:15.602 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:15.619 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.619+0000 7f30ff64b380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:15.620 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.619+0000 7f30ff64b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:15.620 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.619+0000 7f30ff64b380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:45:15.620 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.619+0000 7f30ff64b380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:15.666 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.666+0000 7f30ff64b380 1 bucket sync caught up with source: 2021-09-16T19:45:15.667 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000001.244.12, , ] 2021-09-16T19:45:15.667 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000001.244.12, , ] 2021-09-16T19:45:15.667 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.666+0000 7f30ff64b380 0 bucket checkpoint complete 2021-09-16T19:45:15.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:15.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:15.683 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:15.732 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:15.732 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:15.750 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.749+0000 7f80ac6c9380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:15.750 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.749+0000 7f80ac6c9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:15.750 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.749+0000 7f80ac6c9380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:45:15.751 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.749+0000 7f80ac6c9380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:15.781 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.781+0000 7f80ac6c9380 1 bucket sync caught up with source: 2021-09-16T19:45:15.782 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000001.281.12, , ] 2021-09-16T19:45:15.782 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000001.281.12, , ] 2021-09-16T19:45:15.782 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:15.781+0000 7f80ac6c9380 0 bucket checkpoint complete 2021-09-16T19:45:15.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:15.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:15.799 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:15.847 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:15.848 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:15.865 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.863+0000 7fed15ccb380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:15.865 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.863+0000 7fed15ccb380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:15.865 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.863+0000 7fed15ccb380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:45:15.865 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.863+0000 7fed15ccb380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:15.889 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.888+0000 7fed15ccb380 1 bucket sync caught up with source: 2021-09-16T19:45:15.889 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000001.281.12, , ] 2021-09-16T19:45:15.890 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000001.281.12, , ] 2021-09-16T19:45:15.890 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:15.888+0000 7fed15ccb380 0 bucket checkpoint complete 2021-09-16T19:45:15.903 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-24 zones={test-zone2, test-zone1} 2021-09-16T19:45:15.909 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:15.959 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:15.960 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:15.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:16.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:16.013 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-24 zones={test-zone2, test-zone1} 2021-09-16T19:45:16.014 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-24 zones={test-zone3, test-zone1} 2021-09-16T19:45:16.018 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:16.063 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:16.064 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:16.066 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:16.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:16.121 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-24 zones={test-zone3, test-zone1} 2021-09-16T19:45:16.122 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-24 zones={test-zone3, test-zone2} 2021-09-16T19:45:16.126 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:16.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:16.172 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:16.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:16.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:16.229 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-24 zones={test-zone3, test-zone2} 2021-09-16T19:45:16.230 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:16.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:16.231 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:16.280 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:16.281 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:16.298 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:16.296+0000 7f34f22e9380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:16.298 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:16.296+0000 7f34f22e9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:16.298 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:16.296+0000 7f34f22e9380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:45:16.299 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:16.296+0000 7f34f22e9380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:16.322 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:16.320+0000 7f34f22e9380 1 bucket sync caught up with source: 2021-09-16T19:45:16.322 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000001.13.12, , ] 2021-09-16T19:45:16.322 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000001.13.12, , ] 2021-09-16T19:45:16.323 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:16.320+0000 7f34f22e9380 0 bucket checkpoint complete 2021-09-16T19:45:16.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:16.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:16.336 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:16.385 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:16.385 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:16.403 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.401+0000 7fbffb5f1380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:16.403 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.401+0000 7fbffb5f1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:16.403 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.401+0000 7fbffb5f1380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:45:16.403 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.401+0000 7fbffb5f1380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:16.428 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.427+0000 7fbffb5f1380 1 bucket sync caught up with source: 2021-09-16T19:45:16.428 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000001.13.12, , ] 2021-09-16T19:45:16.429 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000001.13.12, , ] 2021-09-16T19:45:16.429 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.427+0000 7fbffb5f1380 0 bucket checkpoint complete 2021-09-16T19:45:16.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:16.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:16.450 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:16.499 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:16.500 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:16.517 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:16.515+0000 7ffa79be7380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:16.517 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:16.515+0000 7ffa79be7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:16.517 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:16.515+0000 7ffa79be7380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:45:16.517 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:16.515+0000 7ffa79be7380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:16.541 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:16.539+0000 7ffa79be7380 1 bucket sync caught up with source: 2021-09-16T19:45:16.541 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000001.13.12, , ] 2021-09-16T19:45:16.541 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000001.13.12, , ] 2021-09-16T19:45:16.541 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:16.539+0000 7ffa79be7380 0 bucket checkpoint complete 2021-09-16T19:45:16.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:16.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:16.554 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:16.604 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:16.604 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:16.621 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.620+0000 7ff5f228f380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:16.621 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.620+0000 7ff5f228f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:16.622 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.620+0000 7ff5f228f380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:45:16.622 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.620+0000 7ff5f228f380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:16.657 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.656+0000 7ff5f228f380 1 bucket sync caught up with source: 2021-09-16T19:45:16.657 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000003.688.6, , ] 2021-09-16T19:45:16.658 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000003.688.6, , ] 2021-09-16T19:45:16.658 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.656+0000 7ff5f228f380 0 bucket checkpoint complete 2021-09-16T19:45:16.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:16.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:16.675 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:16.723 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:16.723 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:16.742 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.740+0000 7fd5e54ac380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:16.742 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.740+0000 7fd5e54ac380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:16.743 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.740+0000 7fd5e54ac380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:45:16.743 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.740+0000 7fd5e54ac380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:16.792 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.790+0000 7fd5e54ac380 1 bucket sync caught up with source: 2021-09-16T19:45:16.792 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000003.688.6, , ] 2021-09-16T19:45:16.792 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000003.688.6, , ] 2021-09-16T19:45:16.792 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.790+0000 7fd5e54ac380 0 bucket checkpoint complete 2021-09-16T19:45:16.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:16.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:16.808 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:16.857 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:16.857 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:16.879 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.874+0000 7f63d28fc380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:16.879 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.874+0000 7f63d28fc380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:16.880 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.874+0000 7f63d28fc380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:45:16.880 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.874+0000 7f63d28fc380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:16.911 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.910+0000 7f63d28fc380 1 bucket sync caught up with source: 2021-09-16T19:45:16.911 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000001.153.12, , ] 2021-09-16T19:45:16.912 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000001.153.12, , ] 2021-09-16T19:45:16.912 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:16.910+0000 7f63d28fc380 0 bucket checkpoint complete 2021-09-16T19:45:16.924 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:16.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:16.925 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:16.975 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:16.976 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:16.993 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:16.991+0000 7faa72288380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:16.993 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:16.991+0000 7faa72288380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:16.993 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:16.991+0000 7faa72288380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:45:16.993 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:16.991+0000 7faa72288380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:17.017 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:17.015+0000 7faa72288380 1 bucket sync caught up with source: 2021-09-16T19:45:17.017 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000001.153.12, , ] 2021-09-16T19:45:17.018 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000001.153.12, , ] 2021-09-16T19:45:17.018 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:17.015+0000 7faa72288380 0 bucket checkpoint complete 2021-09-16T19:45:17.029 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-25 zones={test-zone2, test-zone1} 2021-09-16T19:45:17.034 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:17.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:17.080 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:17.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:17.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:17.137 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-25 zones={test-zone2, test-zone1} 2021-09-16T19:45:17.137 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-25 zones={test-zone3, test-zone1} 2021-09-16T19:45:17.142 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:17.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:17.187 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:17.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:17.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:17.245 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-25 zones={test-zone3, test-zone1} 2021-09-16T19:45:17.245 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-25 zones={test-zone3, test-zone2} 2021-09-16T19:45:17.249 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:17.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:17.300 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:17.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:17.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:17.353 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-25 zones={test-zone3, test-zone2} 2021-09-16T19:45:17.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:17.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:17.355 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:17.405 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:17.406 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:17.423 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:17.421+0000 7f7772275380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:17.423 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:17.421+0000 7f7772275380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:17.423 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:17.421+0000 7f7772275380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:45:17.423 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:17.421+0000 7f7772275380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:17.447 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:17.445+0000 7f7772275380 1 bucket sync caught up with source: 2021-09-16T19:45:17.447 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000001.9.12, , ] 2021-09-16T19:45:17.447 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000001.9.12, , ] 2021-09-16T19:45:17.448 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:17.445+0000 7f7772275380 0 bucket checkpoint complete 2021-09-16T19:45:17.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:17.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:17.461 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:17.511 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:17.512 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:17.529 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.528+0000 7fbc65937380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:17.529 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.528+0000 7fbc65937380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:17.529 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.528+0000 7fbc65937380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:45:17.530 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.528+0000 7fbc65937380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:17.563 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.562+0000 7fbc65937380 1 bucket sync caught up with source: 2021-09-16T19:45:17.564 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000001.9.12, , ] 2021-09-16T19:45:17.564 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000001.9.12, , ] 2021-09-16T19:45:17.564 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.562+0000 7fbc65937380 0 bucket checkpoint complete 2021-09-16T19:45:17.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:17.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:17.586 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:17.636 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:17.636 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:17.653 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:17.652+0000 7f8db7e67380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:17.654 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:17.652+0000 7f8db7e67380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:17.654 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:17.652+0000 7f8db7e67380 1 Zone: test-zone4 (c6342467-0ed0-40c7-a5ab-491a56113d8e) 2021-09-16T19:45:17.654 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:17.652+0000 7f8db7e67380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:17.677 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:17.675+0000 7f8db7e67380 1 bucket sync caught up with source: 2021-09-16T19:45:17.677 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000001.9.12, , ] 2021-09-16T19:45:17.677 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000001.9.12, , ] 2021-09-16T19:45:17.678 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:17.675+0000 7f8db7e67380 0 bucket checkpoint complete 2021-09-16T19:45:17.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:17.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:17.690 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:17.739 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:17.740 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:17.757 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.756+0000 7f2392a19380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:17.757 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.756+0000 7f2392a19380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:17.758 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.756+0000 7f2392a19380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:45:17.758 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.756+0000 7f2392a19380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:17.796 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.795+0000 7f2392a19380 1 bucket sync caught up with source: 2021-09-16T19:45:17.796 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000001.233.12, , ] 2021-09-16T19:45:17.796 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000001.233.12, , ] 2021-09-16T19:45:17.797 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.795+0000 7f2392a19380 0 bucket checkpoint complete 2021-09-16T19:45:17.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:17.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:17.819 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:17.869 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:17.870 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:17.887 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.886+0000 7f5913a83380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:17.888 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.886+0000 7f5913a83380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:17.888 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.886+0000 7f5913a83380 1 Zone: test-zone3 (8df190b8-e36d-4ab9-bca1-30bba5bb5ec5) 2021-09-16T19:45:17.888 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.886+0000 7f5913a83380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:17.926 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.924+0000 7f5913a83380 1 bucket sync caught up with source: 2021-09-16T19:45:17.926 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000001.233.12, , ] 2021-09-16T19:45:17.927 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000001.233.12, , ] 2021-09-16T19:45:17.927 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:17.924+0000 7f5913a83380 0 bucket checkpoint complete 2021-09-16T19:45:17.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:17.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:17.946 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket htrfwo-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:17.995 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:17.996 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:18.013 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:18.012+0000 7f2675115380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:18.014 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:18.012+0000 7f2675115380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:18.014 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:18.012+0000 7f2675115380 1 Zone: test-zone1 (6395206f-e97e-464f-b7bd-c10485d78bf5) 2021-09-16T19:45:18.015 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:18.012+0000 7f2675115380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:18.040 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:18.038+0000 7f2675115380 1 bucket sync caught up with source: 2021-09-16T19:45:18.040 INFO:teuthology.orchestra.run.smithi109.stderr: local status: [, , , , , , , , 00000000003.447.6, , ] 2021-09-16T19:45:18.041 INFO:teuthology.orchestra.run.smithi109.stderr: remote markers: [, , , , , , , , 00000000003.447.6, , ] 2021-09-16T19:45:18.041 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T19:45:18.038+0000 7f2675115380 0 bucket checkpoint complete 2021-09-16T19:45:18.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:18.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'htrfwo-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-16T19:45:18.054 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket htrfwo-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-16T19:45:18.104 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:18.105 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:18.122 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:18.120+0000 7fc624120380 1 Realm: test-realm (24db56c9-e102-4b9e-a672-6434941d0bb3) 2021-09-16T19:45:18.122 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:18.120+0000 7fc624120380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-16T19:45:18.123 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:18.120+0000 7fc624120380 1 Zone: test-zone2 (ceba8a70-b881-4808-8acf-e8ddfa5f6daf) 2021-09-16T19:45:18.123 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:18.120+0000 7fc624120380 1 using period configuration: db0b3d90-59a0-4d0b-99ed-8408dd9cf54f:4 2021-09-16T19:45:18.146 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:18.145+0000 7fc624120380 1 bucket sync caught up with source: 2021-09-16T19:45:18.147 INFO:teuthology.orchestra.run.smithi118.stderr: local status: [, , , , , , , , 00000000003.447.6, , ] 2021-09-16T19:45:18.147 INFO:teuthology.orchestra.run.smithi118.stderr: remote markers: [, , , , , , , , 00000000003.447.6, , ] 2021-09-16T19:45:18.147 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T19:45:18.145+0000 7fc624120380 0 bucket checkpoint complete 2021-09-16T19:45:18.159 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-26 zones={test-zone2, test-zone1} 2021-09-16T19:45:18.164 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:18.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:18.212 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:18.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:18.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:18.269 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-26 zones={test-zone2, test-zone1} 2021-09-16T19:45:18.269 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-26 zones={test-zone3, test-zone1} 2021-09-16T19:45:18.274 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:18.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:18.319 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:18.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:18.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:18.373 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-26 zones={test-zone3, test-zone1} 2021-09-16T19:45:18.374 INFO:tasks.rgw_multi.zone_rados:comparing bucket=htrfwo-26 zones={test-zone3, test-zone2} 2021-09-16T19:45:18.379 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-16T19:45:18.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:18.424 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-16T19:45:18.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-16T19:45:18.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-16T19:45:18.478 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=htrfwo-26 zones={test-zone3, test-zone2} 2021-09-16T19:45:18.478 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2021-09-16T19:45:18.479 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-27 2021-09-16T19:45:18.495 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=htrfwo-28 2021-09-16T19:45:18.527 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=htrfwo-29 2021-09-16T19:45:18.715 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2021-09-16T19:45:18.716 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-30 2021-09-16T19:45:18.729 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=htrfwo-31 2021-09-16T19:45:18.754 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=htrfwo-32 2021-09-16T19:45:18.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2021-09-16T19:45:18.948 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-33 2021-09-16T19:45:18.995 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=htrfwo-34 2021-09-16T19:45:19.087 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=htrfwo-35 2021-09-16T19:45:21.347 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2021-09-16T19:45:21.348 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-36 2021-09-16T19:45:21.366 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=htrfwo-37 2021-09-16T19:45:21.415 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=htrfwo-38 2021-09-16T19:45:21.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:21.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:21.459 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:45:21.502 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:21.503 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:21.554 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:45:21.567 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:45:21.568 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:45:21.568 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:45:21.568 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:45:21.568 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:45:21.568 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:45:21.569 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:45:21.569 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:45:21.569 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:45:21.569 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:45:21.570 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821521.444824_245.1", 2021-09-16T19:45:21.570 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:45:21.444824Z" 2021-09-16T19:45:21.570 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:45:21.570 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:45:21.570 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:45:21.571 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:17.633408Z" 2021-09-16T19:45:21.571 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:45:21.571 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:45:21.572 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:45:21.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:21.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:21.574 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:45:21.656 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:21.656 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:22.000 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:45:22.000 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:45:22.001 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:45:22.001 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:45:22.001 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:45:22.001 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:45:22.002 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:22.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:22.002 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:45:22.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:22.003 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:45:22.003 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:22.003 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:22.003 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:45:22.003 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:22.004 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:45:22.004 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:22.004 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:45:22.004 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:45:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:22.006 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:22.006 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:45:22.006 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:22.006 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:22.007 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:22.008 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:45:22.008 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:22.008 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:22.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:22.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:22.009 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:45:22.009 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:22.010 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:22.010 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:45:22.010 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:22.010 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:22.011 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:22.011 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:45:22.011 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:22.011 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:22.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:22.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:22.012 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:45:22.012 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:22.012 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:22.013 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:45:22.013 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:22.013 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:22.013 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:22.013 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:45:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:45:22.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:22.015 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:45:22.015 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:45:22.015 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:45:22.015 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:22.016 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:45:22.016 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.488881_181.1 2021-09-16T19:45:27.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:27.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:27.019 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:45:27.061 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:27.062 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:27.110 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:45:27.110 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:45:27.110 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:45:27.111 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:45:27.111 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:45:27.111 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:45:27.111 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:27.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:27.112 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:45:27.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:27.112 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:45:27.112 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:27.113 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:27.113 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:45:27.113 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:27.113 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:45:27.114 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:27.114 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:45:27.114 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:27.114 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:27.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:27.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:27.115 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:45:27.115 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:27.115 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:27.116 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:45:27.116 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:27.116 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:27.116 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:27.116 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:45:27.117 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:27.117 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:27.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:27.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:27.117 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:45:27.118 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:27.118 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:27.118 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.563375_192.1", 2021-09-16T19:45:27.118 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:27.119 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:27.119 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:27.119 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.563375Z", 2021-09-16T19:45:27.119 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:27.119 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:27.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:27.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:27.120 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:45:27.120 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:27.121 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:27.121 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:45:27.121 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:27.121 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:27.122 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:27.122 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:45:27.122 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:27.122 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:27.122 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:27.123 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:45:27.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:27.123 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:45:27.123 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:45:27.124 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:45:27.124 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:27.124 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:45:27.125 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.563375_192.1 2021-09-16T19:45:32.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:32.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:32.128 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:45:32.203 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:32.204 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:32.342 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:45:32.342 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:45:32.343 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:45:32.343 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:45:32.343 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:45:32.343 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:45:32.344 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:32.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:32.344 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:45:32.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:32.344 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:45:32.345 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:32.345 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:32.345 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:45:32.345 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:32.345 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:45:32.346 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:32.346 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:45:32.346 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:32.346 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:32.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:32.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:32.347 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:45:32.347 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:32.347 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:32.348 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:45:32.348 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:32.348 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:32.348 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:32.348 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:45:32.349 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:32.349 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:32.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:32.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:32.350 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:45:32.350 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:32.350 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:32.350 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:45:32.350 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:32.351 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:32.351 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:32.351 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:45:32.351 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:32.351 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:32.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:32.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:32.352 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:45:32.352 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:32.353 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:32.353 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:45:32.353 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:32.353 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:32.353 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:32.354 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:45:32.354 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:32.354 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:32.354 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:32.355 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:45:32.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:32.355 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:45:32.355 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:45:32.355 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:45:32.356 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:32.356 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:45:32.357 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:45:37.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:37.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:37.360 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:45:37.402 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:37.403 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:37.451 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:45:37.451 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:45:37.452 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:45:37.452 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:45:37.452 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:45:37.452 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:45:37.453 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:37.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:37.453 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:45:37.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:37.454 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:45:37.454 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:37.454 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:37.454 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:45:37.454 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:37.455 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:45:37.455 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:37.455 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:45:37.455 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:37.456 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:37.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:37.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:37.456 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:45:37.456 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:37.457 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:37.457 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:45:37.457 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:37.457 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:37.458 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:37.458 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:45:37.458 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:37.458 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:37.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:37.459 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:37.459 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:45:37.459 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:37.459 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:37.460 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.538756_188.1", 2021-09-16T19:45:37.460 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:37.460 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:37.461 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:37.461 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.538756Z", 2021-09-16T19:45:37.461 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:37.461 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:37.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:37.462 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:37.462 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:45:37.463 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:37.463 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:37.463 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:45:37.463 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:37.464 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:37.464 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:37.464 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:45:37.464 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:37.464 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:37.465 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:37.465 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:45:37.465 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:37.465 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:45:37.465 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:45:37.466 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:45:37.466 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:37.466 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:45:37.467 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.538756_188.1 2021-09-16T19:45:42.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:42.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:42.470 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:45:42.520 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:42.520 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:42.582 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:45:42.582 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:45:42.583 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:45:42.583 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:45:42.583 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:45:42.583 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:45:42.584 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:42.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:42.584 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:45:42.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:42.585 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:45:42.585 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:42.585 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:42.585 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:45:42.585 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:42.586 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:45:42.586 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:42.586 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:45:42.587 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:42.587 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:42.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:42.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:42.587 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:45:42.588 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:42.588 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:42.588 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:45:42.588 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:42.589 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:42.589 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:42.589 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:45:42.589 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:42.589 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:42.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:42.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:42.590 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:45:42.590 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:42.591 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:42.591 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:45:42.591 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:42.591 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:42.591 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:42.592 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:45:42.592 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:42.592 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:42.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:42.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:42.593 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:45:42.593 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:42.593 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:42.593 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:45:42.594 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:42.594 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:42.594 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:42.594 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:45:42.595 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:42.595 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:42.595 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:42.596 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:45:42.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:42.596 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:45:42.596 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:45:42.597 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:45:42.597 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:42.597 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:45:42.598 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:45:43.153 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T19:45:43.152+0000 7f2fcaf3e700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-16T19:45:47.599 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:47.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:47.601 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:45:47.656 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:47.657 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:47.708 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:45:47.709 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:45:47.709 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:45:47.709 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:45:47.709 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:45:47.710 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:45:47.710 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:47.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:47.710 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:45:47.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:47.711 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:45:47.711 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:47.711 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:47.711 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:45:47.712 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:47.712 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:45:47.712 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:47.712 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:45:47.713 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:47.713 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:47.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:47.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:47.713 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:45:47.714 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:47.714 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:47.714 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:45:47.714 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:47.715 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:47.715 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:47.715 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:45:47.715 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:47.715 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:47.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:47.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:47.716 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:45:47.716 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:47.716 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:47.717 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:45:47.717 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:47.717 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:47.717 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:47.718 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:45:47.718 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:47.718 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:47.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:47.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:47.719 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:45:47.719 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:47.719 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:47.719 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:45:47.719 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:47.720 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:47.720 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:47.720 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:45:47.720 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:47.721 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:47.721 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:47.721 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:45:47.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:47.721 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:45:47.722 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:45:47.722 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:45:47.722 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:47.722 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:45:47.724 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.488881_181.1 2021-09-16T19:45:52.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:52.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:52.726 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:45:52.769 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:52.770 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:52.818 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:45:52.818 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:45:52.818 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:45:52.819 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:45:52.819 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:45:52.819 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:45:52.819 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:52.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:52.820 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:45:52.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:52.820 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:45:52.820 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:52.821 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:52.821 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:45:52.821 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:52.821 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:45:52.822 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:52.822 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:45:52.822 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:52.822 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:52.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:52.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:52.823 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:45:52.823 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:52.824 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:52.824 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:45:52.824 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:52.824 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:52.825 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:52.825 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:45:52.825 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:52.825 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:52.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:52.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:52.826 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:45:52.826 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:52.826 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:52.826 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:45:52.827 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:52.827 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:52.827 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:52.827 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:45:52.828 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:52.828 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:52.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:52.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:52.828 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:45:52.829 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:52.829 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:52.829 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:45:52.829 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:52.829 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:52.830 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:52.830 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:45:52.831 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:52.831 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:52.831 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:52.831 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:45:52.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:52.832 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:45:52.832 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:45:52.832 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:45:52.833 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:52.833 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:45:52.834 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.488881_181.1 2021-09-16T19:45:57.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:57.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:45:57.837 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:45:57.882 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:45:57.882 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:45:57.933 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:45:57.933 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:45:57.933 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:45:57.933 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:45:57.934 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:45:57.934 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:45:57.934 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:57.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:57.935 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:45:57.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:57.935 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:45:57.935 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:57.936 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:57.936 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:45:57.936 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:57.936 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:45:57.937 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:57.937 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:45:57.937 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:57.937 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:57.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:57.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:57.938 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:45:57.938 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:57.938 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:57.939 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:45:57.939 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:57.939 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:57.939 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:57.939 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:45:57.940 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:57.940 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:57.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:57.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:57.941 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:45:57.941 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:57.941 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:57.941 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.538756_188.1", 2021-09-16T19:45:57.942 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:57.942 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:57.942 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:57.942 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.538756Z", 2021-09-16T19:45:57.943 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:57.943 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:57.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:57.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:45:57.944 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:45:57.944 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:45:57.945 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:45:57.945 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:45:57.945 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:45:57.945 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:45:57.946 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:45:57.946 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:45:57.946 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:45:57.946 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:57.946 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:57.947 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:45:57.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:45:57.947 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:45:57.947 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:45:57.948 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:45:57.948 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:45:57.948 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:45:57.949 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.538756_188.1 2021-09-16T19:46:02.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:02.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:02.952 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:46:03.030 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:46:03.030 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:46:03.152 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T19:46:03.151+0000 7f2fcaf3e700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-16T19:46:03.351 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:46:03.352 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:46:03.352 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:46:03.352 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:46:03.352 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:46:03.353 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:46:03.353 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:03.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:03.353 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:46:03.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:03.354 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:46:03.354 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:03.354 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:03.354 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:46:03.355 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:03.355 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:46:03.355 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:03.355 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:46:03.356 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:03.356 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:03.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:03.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:03.358 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:46:03.359 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:03.359 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:03.359 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:46:03.359 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:03.360 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:03.360 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:03.360 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:46:03.360 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:03.360 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:03.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:03.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:03.361 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:46:03.361 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:03.362 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:03.362 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:46:03.362 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:03.362 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:03.363 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:03.363 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:46:03.363 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:03.363 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:03.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:03.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:03.364 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:46:03.364 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:03.364 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:03.365 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:46:03.365 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:03.365 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:03.365 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:03.365 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:46:03.366 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:03.366 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:03.366 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:03.366 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:46:03.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:03.367 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:46:03.367 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:46:03.367 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:46:03.367 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:03.368 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:46:03.369 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:46:08.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:08.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:08.372 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:46:08.444 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:46:08.444 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:46:08.508 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:46:08.509 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:46:08.509 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:46:08.509 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:46:08.509 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:46:08.510 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:46:08.510 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:08.510 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:08.510 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:46:08.511 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:08.511 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:46:08.511 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:08.511 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:08.512 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:46:08.512 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:08.512 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:46:08.512 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:08.512 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:46:08.513 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:08.513 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:08.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:08.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:08.513 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:46:08.514 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:08.514 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:08.514 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:46:08.514 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:08.515 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:08.515 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:08.515 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:46:08.515 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:08.515 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:08.516 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:08.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:08.516 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:46:08.516 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:08.517 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:08.517 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821519.105698_224.1", 2021-09-16T19:46:08.517 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:08.517 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:08.517 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:08.518 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:19.105698Z", 2021-09-16T19:46:08.518 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:08.518 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:08.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:08.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:08.519 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:46:08.519 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:08.519 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:08.519 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:46:08.520 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:08.520 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:08.520 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:08.520 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:46:08.520 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:08.521 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:08.521 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:08.521 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:46:08.521 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:08.521 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:46:08.522 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:46:08.522 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:46:08.522 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:08.522 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:46:08.524 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821519.105698_224.1 2021-09-16T19:46:13.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:13.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:13.527 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:46:13.572 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:46:13.573 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:46:13.624 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:46:13.625 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:46:13.625 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:46:13.625 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:46:13.626 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:46:13.626 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:46:13.626 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:13.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:13.627 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:46:13.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:13.627 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:46:13.627 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:13.627 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:13.628 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:46:13.628 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:13.628 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:46:13.629 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:13.629 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:46:13.629 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:13.630 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:13.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:13.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:13.630 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:46:13.630 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:13.631 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:13.631 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:46:13.631 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:13.632 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:13.632 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:13.632 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:46:13.632 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:13.632 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:13.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:13.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:13.633 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:46:13.634 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:13.634 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:13.634 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.492123_183.1", 2021-09-16T19:46:13.634 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:13.634 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:13.635 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:13.635 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.492123Z", 2021-09-16T19:46:13.635 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:13.635 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:13.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:13.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:13.636 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:46:13.636 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:13.636 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:13.637 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:46:13.637 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:13.637 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:13.637 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:13.638 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:46:13.638 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:13.638 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:13.639 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:13.639 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:46:13.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:13.639 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:46:13.640 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:46:13.640 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:46:13.640 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:13.640 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:46:13.642 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.492123_183.1 2021-09-16T19:46:18.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:18.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:18.645 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:46:18.691 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:46:18.691 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:46:18.744 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:46:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:46:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:46:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:46:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:46:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:46:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:46:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:46:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:46:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:46:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:46:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:46:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:46:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:46:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:46:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:46:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:46:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:46:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:46:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:46:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:46:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:46:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:46:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:46:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:18.759 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:46:18.760 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.488881_181.1 2021-09-16T19:46:23.761 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:23.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:23.763 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:46:23.854 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:46:23.854 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:46:24.035 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:46:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:46:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:46:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:46:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:46:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:46:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:46:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:46:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:46:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:46:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:46:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:46:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:46:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:46:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:46:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.542659_189.1", 2021-09-16T19:46:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.542659Z", 2021-09-16T19:46:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:24.046 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:24.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:24.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:24.046 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:46:24.047 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:24.047 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:24.047 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:46:24.047 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:46:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:46:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:46:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:46:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:46:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:24.050 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:46:24.052 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.542659_189.1 2021-09-16T19:46:29.053 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:29.054 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:29.055 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:46:29.097 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:46:29.098 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:46:29.146 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:46:29.146 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:46:29.146 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:46:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:46:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:46:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:46:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:29.148 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:46:29.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:29.148 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:46:29.148 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:29.149 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:29.149 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:46:29.149 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:29.149 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:46:29.150 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:29.150 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:46:29.150 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:29.150 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:46:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:29.152 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:46:29.152 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:29.152 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:29.152 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:46:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:46:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:46:29.155 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:29.155 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:29.155 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:29.155 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:46:29.156 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:29.156 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:29.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:29.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:29.156 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:46:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:46:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:46:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:46:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:46:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:46:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:46:29.160 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:29.160 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:46:29.161 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:46:34.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:34.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:34.164 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:46:34.211 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:46:34.212 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:46:34.267 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:46:34.267 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:46:34.267 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:46:34.268 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:46:34.268 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:46:34.268 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:46:34.268 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:34.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:34.269 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:46:34.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:34.269 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:46:34.269 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:34.270 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:34.270 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:46:34.270 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:34.270 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:46:34.271 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:34.271 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:46:34.271 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:34.271 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:34.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:34.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:34.272 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:46:34.272 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:34.273 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:34.273 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:46:34.273 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:34.273 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:34.274 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:34.274 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:46:34.274 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:34.274 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:34.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:34.275 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:34.275 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:46:34.275 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:34.275 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:34.276 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:46:34.276 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:34.276 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:34.276 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:34.276 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:46:34.277 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:34.277 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:34.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:34.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:34.278 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:46:34.278 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:34.278 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:34.278 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:46:34.279 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:34.279 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:34.279 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:34.279 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:46:34.280 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:34.280 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:34.280 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:34.280 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:46:34.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:34.281 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:46:34.281 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:46:34.281 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:46:34.281 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:34.282 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:46:34.283 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.488881_181.1 2021-09-16T19:46:39.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:39.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:39.286 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:46:39.385 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:46:39.385 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:46:39.582 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:46:39.583 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:46:39.583 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:46:39.583 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:46:39.583 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:46:39.583 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:46:39.584 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:39.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:39.584 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:46:39.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:39.585 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:46:39.585 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:39.585 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:39.585 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:46:39.586 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:39.586 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:46:39.586 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:39.586 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:46:39.586 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:39.587 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:39.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:39.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:39.587 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:46:39.587 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:39.588 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:39.588 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:46:39.588 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:39.588 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:39.589 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:39.589 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:46:39.589 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:39.589 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:39.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:39.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:39.590 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:46:39.590 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:39.590 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:39.590 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:46:39.591 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:39.591 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:39.591 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:39.591 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:46:39.591 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:39.592 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:39.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:39.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:39.592 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:46:39.593 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:39.593 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:39.593 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:46:39.593 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:39.593 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:39.594 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:39.594 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:46:39.594 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:39.594 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:39.594 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:39.595 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:46:39.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:39.595 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:46:39.596 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:46:39.596 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:46:39.596 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:39.596 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:46:39.597 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:46:44.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:44.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:44.600 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:46:44.642 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:46:44.643 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:46:44.692 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:46:44.693 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:46:44.693 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:46:44.693 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:46:44.693 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:46:44.693 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:46:44.694 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:44.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:44.694 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:46:44.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:44.695 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:46:44.695 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:44.695 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:44.695 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:46:44.695 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:46:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:46:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:46:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:46:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:46:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:46:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.563375_192.1", 2021-09-16T19:46:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.563375Z", 2021-09-16T19:46:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:46:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:46:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:46:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:46:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:46:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:46:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:46:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:44.706 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:46:44.707 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.563375_192.1 2021-09-16T19:46:49.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:49.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:49.711 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:46:49.761 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:46:49.761 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:46:49.816 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:46:49.817 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:46:49.817 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:46:49.817 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:46:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:46:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:46:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:49.818 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:46:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:46:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:49.819 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:46:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:46:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:49.820 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:46:49.821 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:46:49.822 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:46:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:49.823 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:46:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:49.824 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:46:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:49.825 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.725649_200.1", 2021-09-16T19:46:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:49.826 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.725649Z", 2021-09-16T19:46:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:49.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:49.828 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:46:49.828 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:49.828 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:49.828 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:46:49.828 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:46:49.829 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:46:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:49.830 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:46:49.831 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:46:49.831 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:46:49.831 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:49.831 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:46:49.833 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.725649_200.1 2021-09-16T19:46:54.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:54.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:46:54.835 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:46:54.914 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:46:54.915 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:46:55.263 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:46:55.263 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:46:55.263 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:46:55.264 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:46:55.264 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:46:55.264 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:46:55.264 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:55.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:55.265 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:46:55.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:55.265 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:46:55.265 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:55.266 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:55.266 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:46:55.266 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:55.266 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:46:55.267 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:55.268 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:46:55.268 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:55.269 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:55.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:55.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:55.269 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:46:55.270 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:55.270 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:55.270 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:46:55.270 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:55.270 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:55.271 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:55.271 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:46:55.271 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:55.271 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:55.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:55.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:55.272 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:46:55.272 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:55.272 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:55.273 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:46:55.273 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:55.273 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:55.273 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:55.274 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:46:55.274 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:55.274 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:55.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:55.275 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:46:55.275 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:46:55.275 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:46:55.275 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:46:55.276 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:46:55.276 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:46:55.276 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:46:55.277 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:46:55.277 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:46:55.277 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:46:55.277 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:55.278 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:55.278 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:46:55.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:46:55.278 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:46:55.278 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:46:55.279 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:46:55.279 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:46:55.279 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:46:55.281 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:47:00.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:00.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:00.283 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:47:00.326 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:47:00.327 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:47:00.374 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:47:00.375 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:47:00.375 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:47:00.375 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:47:00.376 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:47:00.376 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:47:00.376 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:00.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:00.376 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:47:00.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:00.377 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:47:00.377 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:00.377 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:00.378 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:47:00.378 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:00.378 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:47:00.378 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:00.378 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:47:00.379 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:00.379 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:00.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:00.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:00.380 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:47:00.380 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:00.380 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:00.380 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:47:00.381 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:00.381 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:00.381 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:00.381 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:47:00.381 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:00.382 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:00.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:00.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:00.382 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:47:00.383 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:00.383 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:00.383 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:47:00.383 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:00.383 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:00.384 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:00.384 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:47:00.384 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:00.384 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:00.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:00.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:00.385 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:47:00.385 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:00.385 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:00.386 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:47:00.386 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:00.386 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:00.387 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:00.387 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:47:00.387 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:00.388 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:00.388 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:00.388 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:47:00.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:00.389 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:47:00.389 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:47:00.389 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:47:00.389 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:00.389 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:47:00.391 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:47:05.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:05.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:05.394 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:47:05.445 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:47:05.445 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:47:05.507 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:47:05.507 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:47:05.508 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:47:05.508 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:47:05.508 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:47:05.508 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:47:05.509 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:05.509 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:05.509 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:47:05.509 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:05.509 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:47:05.510 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:05.510 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:05.510 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:47:05.510 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:05.511 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:47:05.511 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:05.511 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:47:05.511 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:05.512 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:05.512 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:05.512 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:05.512 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:47:05.513 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:05.513 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:05.513 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:47:05.513 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:05.513 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:05.514 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:05.514 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:47:05.514 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:05.514 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:05.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:05.515 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:05.515 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:47:05.515 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:05.515 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:05.516 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:47:05.516 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:05.516 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:05.516 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:05.517 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:47:05.517 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:05.517 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:05.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:05.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:05.518 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:47:05.518 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:05.518 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:05.518 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:47:05.519 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:05.519 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:05.519 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:05.519 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:47:05.519 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:05.520 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:05.520 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:05.520 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:47:05.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:05.520 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:47:05.521 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:47:05.521 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:47:05.521 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:05.521 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:47:05.523 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:47:10.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:10.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:10.525 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:47:10.590 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:47:10.590 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:47:11.009 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:47:11.009 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:47:11.010 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:47:11.010 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:47:11.010 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:47:11.010 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:47:11.011 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:11.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:11.011 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:47:11.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:11.011 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:47:11.012 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:11.012 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:11.012 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:47:11.012 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:11.012 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:47:11.013 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:11.013 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:47:11.013 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:11.013 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:11.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:11.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:11.014 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:47:11.014 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:11.015 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:11.015 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:47:11.015 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:11.015 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:11.015 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:11.016 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:47:11.016 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:11.016 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:11.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:11.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:11.017 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:47:11.017 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:11.017 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:11.017 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:47:11.018 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:11.018 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:11.018 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:11.018 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:47:11.018 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:11.019 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:11.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:11.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:11.019 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:47:11.019 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:11.020 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:11.020 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:47:11.020 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:11.020 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:11.020 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:11.021 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:47:11.021 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:11.021 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:11.021 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:11.021 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:47:11.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:11.022 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:47:11.022 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:47:11.022 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:47:11.023 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:11.023 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:47:11.024 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:47:16.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:16.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:16.027 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:47:16.101 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:47:16.101 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:47:16.216 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:47:16.216 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:47:16.216 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:47:16.217 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:47:16.217 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:47:16.217 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:47:16.217 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:16.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:16.218 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:47:16.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:16.219 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:47:16.219 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:16.219 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:16.219 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:47:16.219 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:16.220 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:47:16.220 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:16.220 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:47:16.220 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:16.220 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:16.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:16.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:16.221 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:47:16.221 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:16.221 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:16.222 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:47:16.222 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:16.222 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:16.222 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:16.223 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:47:16.223 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:16.223 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:16.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:16.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:16.224 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:47:16.224 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:16.224 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:16.224 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:47:16.224 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:16.225 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:16.225 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:16.225 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:47:16.225 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:16.225 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:16.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:16.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:16.226 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:47:16.227 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:16.227 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:16.227 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:47:16.227 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:16.228 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:16.228 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:16.228 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:47:16.228 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:16.228 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:16.229 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:16.229 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:47:16.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:16.229 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:47:16.230 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:47:16.230 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:47:16.230 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:16.230 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:47:16.232 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.488881_181.1 2021-09-16T19:47:21.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:21.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:21.235 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:47:21.282 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:47:21.282 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:47:21.333 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:47:21.333 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:47:21.334 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:47:21.334 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:47:21.334 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:47:21.334 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:47:21.335 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:21.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:21.335 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:47:21.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:21.335 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:47:21.336 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:21.336 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:21.336 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:47:21.336 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:21.336 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:47:21.337 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:21.337 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:47:21.337 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:21.337 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:21.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:21.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:21.338 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:47:21.338 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:21.338 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:21.339 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:47:21.339 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:21.339 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:21.339 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:21.339 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:47:21.340 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:21.340 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:21.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:21.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:21.341 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:47:21.341 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:21.341 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:21.341 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:47:21.341 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:21.342 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:21.342 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:21.342 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:47:21.342 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:21.343 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:21.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:21.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:21.343 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:47:21.343 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:21.344 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:21.344 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:47:21.344 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:21.344 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:21.344 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:21.345 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:47:21.345 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:21.345 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:21.345 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:21.345 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:47:21.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:21.346 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:47:21.346 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:47:21.346 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:47:21.347 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:21.347 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:47:21.348 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:47:26.349 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:26.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:26.351 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:47:26.399 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:47:26.399 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:47:26.453 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:47:26.453 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:47:26.453 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:47:26.454 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:47:26.454 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:47:26.454 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:47:26.454 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:26.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:26.455 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:47:26.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:26.455 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:47:26.456 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:26.456 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:26.456 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:47:26.456 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:26.457 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:47:26.457 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:26.457 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:47:26.457 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:26.458 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:26.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:26.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:26.458 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:47:26.459 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:26.459 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:26.459 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:47:26.459 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:26.460 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:26.460 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:26.460 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:47:26.460 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:26.461 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:26.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:26.461 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:26.461 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:47:26.462 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:26.462 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:26.462 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:47:26.462 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:26.462 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:26.463 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:26.463 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:47:26.463 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:26.463 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:26.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:26.464 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:26.464 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:47:26.464 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:26.465 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:26.465 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:47:26.465 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:26.465 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:26.465 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:26.466 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:47:26.466 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:26.466 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:26.466 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:26.467 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:47:26.467 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:26.467 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:47:26.467 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:47:26.468 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:47:26.468 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:26.468 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:47:26.469 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:47:31.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:31.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:31.472 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:47:31.563 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:47:31.563 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:47:31.870 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:47:31.870 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:47:31.870 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:47:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:47:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:47:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:47:31.871 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:47:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:31.872 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:47:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:47:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:31.873 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:47:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:47:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:31.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:47:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:31.875 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:47:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:31.876 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:47:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:31.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:47:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:47:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:47:31.879 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:31.880 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:31.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:31.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:31.880 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:47:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:47:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:31.881 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:31.882 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:31.882 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:47:31.882 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:31.882 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:47:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:47:31.883 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:47:31.884 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:47:31.884 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:31.884 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:47:31.885 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:47:36.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:36.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:36.888 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:47:36.960 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:47:36.960 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:47:37.029 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:47:37.030 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:47:37.030 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:47:37.030 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:47:37.030 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:47:37.031 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:47:37.031 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:37.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:37.031 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:47:37.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:37.032 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:47:37.032 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:37.032 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:37.032 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:47:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:47:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:47:37.033 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:37.034 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:47:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:47:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:37.035 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:47:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:37.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:47:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:37.037 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:47:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:37.038 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:47:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:37.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:47:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:47:37.040 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:47:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:37.041 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:37.042 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:37.042 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:47:37.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:37.042 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:47:37.042 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:47:37.043 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:47:37.043 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:37.043 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:47:37.044 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.543574_190.1 2021-09-16T19:47:38.040 INFO:tasks.rgw.c1.client.1.smithi109.stdout:2021-09-16T19:47:38.039+0000 7f2506e7c700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-16T19:47:42.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:42.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:42.047 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:47:42.090 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:47:42.090 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:47:42.138 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:47:42.139 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:47:42.139 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:47:42.139 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:47:42.140 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:47:42.140 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:47:42.140 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:42.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:42.140 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:47:42.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:42.141 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:47:42.141 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:42.141 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:42.142 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:47:42.142 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:42.142 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:47:42.142 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:42.142 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:47:42.143 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:42.143 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:42.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:42.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:42.143 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:47:42.144 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:42.144 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:42.144 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:47:42.144 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:42.145 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:42.145 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:42.145 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:47:42.145 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:42.145 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:42.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:42.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:42.146 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:47:42.146 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:42.146 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:42.147 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:47:42.147 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:42.147 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:42.147 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:42.147 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:47:42.148 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:42.148 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:42.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:42.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:42.149 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:47:42.149 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:42.149 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:42.149 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:47:42.149 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:42.150 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:42.150 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:42.150 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:47:42.150 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:42.151 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:42.151 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:42.151 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:47:42.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:42.152 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:47:42.152 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:47:42.152 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:47:42.152 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:42.152 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:47:42.154 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:47:47.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:47.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:47.156 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:47:47.204 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:47:47.205 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:47:47.259 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:47:47.259 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:47:47.260 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:47:47.260 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:47:47.260 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:47:47.260 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:47:47.260 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:47.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:47.261 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:47:47.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:47.261 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:47:47.262 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:47.262 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:47.262 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:47:47.262 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:47.262 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:47:47.263 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:47.263 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:47:47.263 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:47.263 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:47.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:47.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:47.264 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:47:47.264 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:47.264 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:47.265 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:47:47.265 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:47.265 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:47.265 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:47.265 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:47:47.266 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:47.266 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:47.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:47.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:47.267 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:47:47.267 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:47.267 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:47.267 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:47:47.267 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:47.268 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:47.268 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:47.268 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:47:47.268 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:47.269 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:47.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:47.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:47.269 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:47:47.269 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:47.270 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:47.270 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:47:47.270 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:47.270 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:47.270 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:47.271 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:47:47.271 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:47.271 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:47.271 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:47.272 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:47:47.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:47.272 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:47:47.272 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:47:47.272 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:47:47.273 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:47.273 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:47:47.274 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.513492_186.1 2021-09-16T19:47:52.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:52.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:52.277 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:47:52.360 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:47:52.361 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:47:52.646 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:47:52.647 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:47:52.647 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:47:52.647 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:47:52.647 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:47:52.648 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:47:52.648 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:52.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:52.648 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:47:52.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:52.649 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:47:52.649 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:52.649 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:52.649 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:47:52.650 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:52.650 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:47:52.650 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:52.650 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:47:52.651 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:52.651 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:52.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:52.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:52.651 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:47:52.652 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:52.652 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:52.652 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:47:52.652 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:52.652 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:52.653 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:52.653 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:47:52.653 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:52.653 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:52.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:52.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:52.654 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:47:52.654 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:52.654 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:52.655 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:47:52.655 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:52.655 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:52.655 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:52.655 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:47:52.656 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:52.656 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:52.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:52.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:52.656 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:47:52.657 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:52.657 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:52.657 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:47:52.657 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:52.657 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:52.658 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:52.658 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:47:52.658 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:52.658 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:52.659 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:52.659 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:47:52.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:52.659 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:47:52.659 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:47:52.660 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:47:52.660 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:52.660 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:47:52.661 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.488881_181.1 2021-09-16T19:47:57.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:57.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:47:57.664 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:47:57.736 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:47:57.736 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:47:57.826 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:47:57.827 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:47:57.827 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:47:57.827 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:47:57.827 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:47:57.827 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:47:57.828 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:57.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:57.828 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:47:57.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:57.828 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:47:57.829 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:57.829 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:57.829 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:47:57.829 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:57.829 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:47:57.830 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:57.830 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:47:57.830 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:57.830 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:57.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:57.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:57.831 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:47:57.831 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:57.831 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:57.832 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:47:57.832 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:57.832 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:57.832 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:57.832 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:47:57.833 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:57.833 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:57.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:57.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:57.833 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:47:57.834 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:57.834 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:57.834 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:47:57.834 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:57.834 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:57.835 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:57.835 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:47:57.835 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:57.835 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:57.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:57.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:47:57.836 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:47:57.836 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:47:57.836 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:47:57.837 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:47:57.837 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:47:57.837 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:47:57.837 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:47:57.837 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:47:57.838 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:47:57.838 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:57.838 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:57.838 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:47:57.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:47:57.839 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:47:57.839 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:47:57.839 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:47:57.839 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:47:57.840 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:47:57.841 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.513492_186.1 2021-09-16T19:47:57.984 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2021-09-16T19:47:57.983+0000 7f0df1560700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-16T19:48:02.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:02.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:02.843 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:48:02.886 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:48:02.886 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:48:02.935 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:48:02.935 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:48:02.935 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:48:02.935 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:48:02.936 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:48:02.936 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:48:02.936 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:02.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:02.936 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:48:02.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:02.937 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:48:02.937 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:02.937 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:02.937 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:48:02.938 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:02.938 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:48:02.938 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:02.938 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:48:02.938 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:02.939 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:02.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:02.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:02.939 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:48:02.939 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:02.940 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:02.940 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:48:02.940 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:02.940 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:02.940 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:02.941 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:48:02.941 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:02.941 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:02.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:02.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:02.942 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:48:02.942 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:02.942 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:02.942 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:48:02.943 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:02.943 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:02.943 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:02.943 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:48:02.943 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:02.944 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:02.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:02.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:02.944 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:48:02.944 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:02.945 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:02.945 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:48:02.945 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:02.945 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:02.945 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:02.946 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:48:02.946 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:02.946 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:02.946 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:02.946 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:48:02.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:02.947 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:48:02.947 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:48:02.947 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:48:02.947 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:02.948 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:48:02.949 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.513492_186.1 2021-09-16T19:48:07.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:07.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:07.951 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:48:07.999 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:48:07.999 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:48:08.055 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:48:08.055 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:48:08.055 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:48:08.056 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:48:08.056 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:48:08.056 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:48:08.056 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:08.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:08.057 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:48:08.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:08.057 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:48:08.058 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:08.058 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:08.058 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:48:08.058 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:08.058 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:48:08.059 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:08.059 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:48:08.059 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:08.059 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:08.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:08.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:08.060 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:48:08.060 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:08.060 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:08.061 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:48:08.061 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:08.061 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:08.061 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:08.062 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:48:08.062 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:08.062 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:08.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:08.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:08.063 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:48:08.063 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:08.063 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:08.063 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.508383_184.1", 2021-09-16T19:48:08.064 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:08.064 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:08.064 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:08.064 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.508383Z", 2021-09-16T19:48:08.065 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:08.065 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:08.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:08.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:08.065 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:48:08.066 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:08.066 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:08.066 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:48:08.066 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:08.067 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:08.067 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:08.067 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:48:08.067 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:08.067 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:08.068 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:08.068 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:48:08.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:08.068 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:48:08.069 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:48:08.069 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:48:08.069 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:08.069 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:48:08.070 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.508383_184.1 2021-09-16T19:48:13.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:13.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:13.073 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:48:13.160 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:48:13.160 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:48:13.547 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:48:13.548 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:48:13.548 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:48:13.548 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:48:13.549 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:48:13.549 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:48:13.549 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:13.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:13.550 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:48:13.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:13.550 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:48:13.550 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:13.550 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:13.551 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:48:13.551 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:13.551 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:48:13.551 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:13.552 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:48:13.552 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:13.552 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:13.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:13.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:13.553 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:48:13.553 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:13.553 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:13.553 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:48:13.554 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:13.554 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:13.554 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:13.554 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:48:13.554 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:13.555 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:13.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:13.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:13.555 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:48:13.556 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:13.556 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:13.556 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.563375_192.1", 2021-09-16T19:48:13.556 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:13.556 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:13.557 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:13.557 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.563375Z", 2021-09-16T19:48:13.557 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:13.557 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:13.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:13.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:13.558 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:48:13.558 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:13.558 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:13.558 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:48:13.559 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:13.559 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:13.559 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:13.559 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:48:13.560 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:13.560 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:13.560 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:13.560 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:48:13.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:13.561 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:48:13.561 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:48:13.561 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:48:13.561 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:13.562 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:48:13.563 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.563375_192.1 2021-09-16T19:48:18.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:18.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:18.565 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:48:18.630 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:48:18.631 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:48:18.744 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:48:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:48:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:48:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:48:18.745 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:48:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:48:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:18.746 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:48:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:48:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:18.747 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:48:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:48:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:18.748 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:48:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:18.749 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:48:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:48:18.750 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:48:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:18.751 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:48:18.752 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.542659_189.1", 2021-09-16T19:48:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:18.753 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.542659Z", 2021-09-16T19:48:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:18.754 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:48:18.755 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:48:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:18.756 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:48:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:18.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:48:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:18.758 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:48:18.759 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:48:18.759 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:48:18.759 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:18.759 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:48:18.761 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.542659_189.1 2021-09-16T19:48:23.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:23.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:23.764 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:48:23.807 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:48:23.807 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:48:23.855 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:48:23.856 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:48:23.856 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:48:23.856 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:48:23.856 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:48:23.857 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:48:23.857 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:23.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:23.857 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:48:23.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:23.858 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:48:23.858 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:23.858 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:23.858 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:48:23.859 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:23.859 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:48:23.859 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:23.859 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:48:23.859 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:23.860 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:23.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:23.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:23.860 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:48:23.860 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:23.861 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:23.861 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:48:23.861 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:23.861 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:23.861 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:23.862 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:48:23.862 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:23.862 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:23.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:23.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:23.863 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:48:23.863 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:23.863 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:23.863 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:48:23.863 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:23.864 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:23.864 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:23.864 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:48:23.864 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:23.864 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:23.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:23.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:23.865 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:48:23.865 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:23.865 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:23.866 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:48:23.866 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:23.866 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:23.866 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:23.866 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:48:23.867 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:23.867 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:23.867 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:23.867 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:48:23.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:23.868 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:48:23.868 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:48:23.868 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:48:23.868 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:23.868 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:48:23.869 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:48:28.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:28.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:28.872 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:48:28.922 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:48:28.922 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:48:28.983 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:48:28.984 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:48:28.984 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:48:28.984 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:48:28.984 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:48:28.985 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:48:28.985 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:28.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:28.985 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:48:28.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:28.986 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:48:28.986 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:28.986 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:28.986 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:48:28.987 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:28.987 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:48:28.987 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:28.987 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:48:28.988 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:28.988 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:28.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:28.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:28.988 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:48:28.989 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:28.989 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:28.989 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:48:28.989 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:28.989 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:28.990 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:28.990 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:48:28.990 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:28.990 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:28.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:28.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:28.991 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:48:28.991 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:28.991 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:28.992 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:48:28.992 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:28.992 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:28.992 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:28.993 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:48:28.993 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:28.993 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:28.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:28.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:28.994 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:48:28.995 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:28.995 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:28.995 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:48:28.995 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:28.995 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:28.996 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:28.996 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:48:28.996 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:28.996 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:28.997 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:28.997 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:48:28.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:28.997 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:48:28.997 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:48:28.998 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:48:28.998 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:28.998 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:48:28.999 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:48:34.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:34.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:34.002 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:48:34.095 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:48:34.095 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:48:34.389 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:48:34.389 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:48:34.389 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:48:34.390 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:48:34.390 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:48:34.390 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:48:34.390 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:34.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:34.391 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:48:34.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:34.391 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:48:34.391 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:34.392 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:34.392 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:48:34.392 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:34.392 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:48:34.392 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:34.392 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:48:34.393 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:34.393 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:34.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:34.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:34.394 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:48:34.394 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:34.394 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:34.394 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:48:34.394 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:34.395 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:34.395 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:34.395 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:48:34.395 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:34.395 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:34.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:34.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:34.396 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:48:34.396 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:34.396 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:34.397 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:48:34.397 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:34.397 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:34.397 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:34.397 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:48:34.398 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:34.398 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:34.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:34.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:34.398 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:48:34.399 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:34.399 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:34.399 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:48:34.399 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:34.399 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:34.400 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:34.400 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:48:34.400 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:34.400 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:34.400 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:34.401 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:48:34.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:34.401 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:48:34.401 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:48:34.401 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:48:34.402 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:34.402 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:48:34.403 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:48:39.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:39.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:39.406 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:48:39.473 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:48:39.474 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:48:39.564 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:48:39.564 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:48:39.564 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:48:39.565 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:48:39.565 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:48:39.565 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:48:39.565 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:39.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:39.566 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:48:39.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:39.566 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:48:39.567 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:39.567 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:39.567 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:48:39.567 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:39.568 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:48:39.568 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:39.568 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:48:39.568 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:39.569 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:39.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:39.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:39.569 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:48:39.570 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:39.570 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:39.570 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:48:39.571 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:39.571 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:39.571 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:39.571 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:48:39.572 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:39.572 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:39.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:39.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:39.573 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:48:39.573 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:39.573 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:39.573 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:48:39.573 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:39.574 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:39.574 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:39.574 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:48:39.574 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:39.574 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:39.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:39.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:39.575 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:48:39.575 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:39.575 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:39.576 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:48:39.576 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:39.576 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:39.576 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:39.576 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:48:39.577 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:39.577 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:39.577 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:39.578 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:48:39.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:39.578 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:48:39.578 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:48:39.579 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:48:39.579 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:39.579 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:48:39.580 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:48:44.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:44.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:44.583 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:48:44.626 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:48:44.626 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:48:44.676 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:48:44.676 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:48:44.677 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:48:44.677 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:48:44.677 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:48:44.677 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:48:44.677 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:44.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:44.678 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:48:44.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:44.678 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:48:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:48:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:48:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:48:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:44.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:44.681 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:48:44.681 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:44.681 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:44.681 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:48:44.682 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:44.682 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:44.682 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:44.682 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:48:44.682 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:44.683 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:44.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:44.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:44.683 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:48:44.683 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:44.684 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:44.684 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.542659_189.1", 2021-09-16T19:48:44.684 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:44.684 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:44.684 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.542659Z", 2021-09-16T19:48:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:48:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:48:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:44.687 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:44.687 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:44.687 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:48:44.687 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:44.687 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:48:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:48:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:48:44.689 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:48:44.689 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:44.689 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:48:44.690 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.542659_189.1 2021-09-16T19:48:49.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:49.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:49.693 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:48:49.740 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:48:49.741 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:48:49.797 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:48:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:48:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:48:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:48:49.798 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:48:49.799 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:48:49.799 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:49.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:49.799 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:48:49.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:48:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:48:49.800 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:48:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:48:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:49.801 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:48:49.802 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:49.803 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:49.803 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:48:49.803 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:49.803 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:49.803 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:49.804 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:48:49.804 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:49.804 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:49.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:49.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:48:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:48:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:48:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:49.806 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:48:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:49.807 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:48:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:49.808 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:48:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:48:49.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:48:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:48:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:48:49.810 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:49.810 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:48:49.812 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.513492_186.1 2021-09-16T19:48:54.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:54.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:48:54.814 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:48:54.917 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:48:54.918 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:48:55.139 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:48:55.140 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:48:55.140 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:48:55.140 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:48:55.141 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:48:55.141 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:48:55.141 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:55.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:55.141 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:48:55.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:55.142 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:48:55.142 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:55.142 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:55.142 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:48:55.143 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:55.143 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:48:55.143 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:55.143 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:48:55.144 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:55.144 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:55.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:55.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:55.144 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:48:55.145 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:55.145 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:55.145 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:48:55.145 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:55.145 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:55.146 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:55.146 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:48:55.146 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:55.146 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:55.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:55.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:55.147 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:48:55.147 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:55.147 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:55.147 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.508383_184.1", 2021-09-16T19:48:55.148 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:55.148 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:55.148 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:55.148 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.508383Z", 2021-09-16T19:48:55.149 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:55.149 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:55.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:55.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:48:55.149 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:48:55.150 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:48:55.150 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:48:55.150 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:48:55.150 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:48:55.151 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:48:55.151 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:48:55.151 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:48:55.151 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:48:55.151 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:55.152 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:55.152 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:48:55.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:48:55.152 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:48:55.153 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:48:55.153 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:48:55.153 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:48:55.153 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:48:55.154 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.508383_184.1 2021-09-16T19:49:00.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:00.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:00.157 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:49:00.227 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:49:00.228 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:49:00.284 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:49:00.284 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:49:00.284 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:49:00.285 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:49:00.286 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:00.287 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:00.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:00.289 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:00.290 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:00.292 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:00.293 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:00.293 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:49:00.293 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:00.293 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:00.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:00.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:00.294 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:49:00.294 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:00.294 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:00.295 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:49:00.295 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:00.295 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:00.295 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:00.295 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:49:00.296 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:00.296 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:00.296 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:00.296 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:49:00.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:00.297 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:49:00.297 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:49:00.297 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:49:00.297 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:00.298 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:49:00.299 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:49:05.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:05.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:05.302 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:49:05.345 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:49:05.346 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:49:05.396 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:49:05.396 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:49:05.396 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:49:05.397 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:49:05.397 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:49:05.397 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:49:05.398 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:05.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:05.398 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:49:05.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:05.399 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:49:05.399 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:05.399 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:05.399 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:49:05.400 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:05.400 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:49:05.400 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:05.400 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:49:05.401 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:05.401 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:05.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:05.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:05.402 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:49:05.402 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:05.402 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:05.403 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:49:05.403 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:05.403 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:05.403 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:05.404 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:49:05.404 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:05.404 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:05.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:05.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:05.405 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:49:05.405 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:05.406 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:05.406 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.768089_209.1", 2021-09-16T19:49:05.406 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:05.406 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:05.407 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:05.407 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.768089Z", 2021-09-16T19:49:05.407 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:05.407 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:05.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:05.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:05.408 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:49:05.408 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:05.408 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:05.409 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:49:05.409 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:05.409 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:05.409 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:05.410 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:49:05.410 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:05.410 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:05.410 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:05.411 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:49:05.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:05.411 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:49:05.411 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:49:05.411 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:49:05.412 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:05.412 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:49:05.413 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.768089_209.1 2021-09-16T19:49:10.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:10.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:10.416 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:49:10.465 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:49:10.465 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:49:10.528 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:49:10.529 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:49:10.529 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:49:10.529 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:49:10.529 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:49:10.530 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:49:10.530 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:10.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:10.530 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:49:10.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:10.531 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:49:10.531 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:10.531 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:10.532 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:49:10.532 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:10.532 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:49:10.532 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:10.533 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:49:10.533 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:10.533 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:10.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:10.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:10.534 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:49:10.534 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:10.534 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:10.535 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:49:10.535 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:10.535 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:10.535 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:10.536 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:49:10.536 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:10.536 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:10.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:10.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:10.537 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:49:10.537 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:10.537 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:10.537 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:49:10.538 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:10.538 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:10.538 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:10.539 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:49:10.539 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:10.539 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:10.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:10.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:10.540 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:49:10.540 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:10.540 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:10.540 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:49:10.541 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:10.541 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:10.541 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:10.541 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:49:10.542 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:10.542 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:10.542 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:10.542 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:49:10.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:10.543 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:49:10.543 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:49:10.543 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:49:10.544 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:10.544 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:49:10.545 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:49:15.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:15.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:15.548 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:49:15.629 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:49:15.629 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:49:15.918 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:49:15.918 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:49:15.918 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:49:15.918 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:49:15.919 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:49:15.919 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:49:15.919 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:15.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:15.920 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:49:15.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:15.920 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:49:15.920 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:15.920 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:15.921 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:49:15.921 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:15.921 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:49:15.921 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:15.922 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:49:15.922 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:15.922 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:15.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:15.923 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:15.923 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:49:15.923 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:15.923 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:15.923 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:49:15.924 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:15.924 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:15.924 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:15.924 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:49:15.924 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:15.925 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:15.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:15.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:15.925 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:49:15.926 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:15.926 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:15.926 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.538756_188.1", 2021-09-16T19:49:15.926 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:15.927 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:15.927 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:15.927 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.538756Z", 2021-09-16T19:49:15.927 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:15.927 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:15.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:15.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:15.928 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:49:15.928 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:15.928 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:15.929 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:49:15.929 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:15.929 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:15.929 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:15.929 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:49:15.930 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:15.930 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:15.930 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:15.930 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:49:15.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:15.931 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:49:15.931 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:49:15.931 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:49:15.931 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:15.932 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:49:15.933 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.538756_188.1 2021-09-16T19:49:20.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:20.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:20.936 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:49:20.981 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:49:20.981 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:49:21.029 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:49:21.030 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:49:21.030 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:49:21.030 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:49:21.031 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:49:21.031 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:49:21.031 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:21.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:21.032 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:49:21.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:21.032 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:49:21.032 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:21.033 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:21.033 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:49:21.033 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:21.033 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:49:21.034 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:21.034 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:49:21.034 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:21.034 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:21.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:21.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:21.035 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:49:21.035 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:21.035 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:21.036 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:49:21.036 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:21.036 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:21.036 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:21.037 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:49:21.037 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:21.037 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:21.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:21.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:21.038 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:49:21.038 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:21.038 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:21.038 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:49:21.039 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:21.039 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:21.039 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:21.039 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:49:21.039 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:21.040 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:21.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:21.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:21.040 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:49:21.041 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:21.041 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:21.041 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:49:21.041 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:21.042 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:21.042 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:21.042 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:49:21.042 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:21.043 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:21.043 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:21.043 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:49:21.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:21.043 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:49:21.044 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:49:21.044 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:49:21.044 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:21.044 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:49:21.046 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.488881_181.1 2021-09-16T19:49:26.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:26.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:26.048 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:49:26.094 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:49:26.095 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:49:26.145 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:49:26.145 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:49:26.146 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:49:26.146 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:49:26.146 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:49:26.146 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:49:26.147 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:26.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:26.147 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:49:26.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:26.148 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:49:26.148 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:26.148 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:26.148 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:49:26.148 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:26.149 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:49:26.149 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:26.149 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:49:26.149 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:26.149 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:26.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:26.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:26.150 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:49:26.150 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:26.150 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:26.151 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:49:26.151 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:26.151 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:26.151 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:26.152 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:49:26.152 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:26.152 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:26.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:26.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:26.153 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:49:26.153 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:26.153 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:26.154 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:49:26.154 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:26.154 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:26.154 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:26.154 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:49:26.155 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:26.155 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:26.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:26.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:26.155 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:49:26.156 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:26.156 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:26.156 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:49:26.156 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:26.157 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:26.157 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:26.157 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:49:26.157 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:26.157 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:26.158 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:26.158 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:49:26.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:26.158 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:49:26.159 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:49:26.159 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:49:26.159 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:26.159 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:49:26.160 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:49:31.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:31.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:31.163 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:49:31.217 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:49:31.217 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:49:31.446 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:49:31.447 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:49:31.447 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:49:31.447 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:49:31.447 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:49:31.448 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:49:31.448 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:31.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:31.448 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:49:31.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:31.449 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:49:31.449 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:31.449 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:31.449 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:49:31.450 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:31.450 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:49:31.450 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:31.450 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:49:31.451 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:31.451 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:31.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:31.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:31.451 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:49:31.452 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:31.452 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:31.452 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:49:31.452 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:31.453 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:31.453 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:31.453 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:49:31.453 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:31.453 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:31.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:31.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:31.454 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:49:31.454 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:31.454 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:31.455 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.492123_183.1", 2021-09-16T19:49:31.455 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:31.455 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:31.455 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:31.456 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.492123Z", 2021-09-16T19:49:31.456 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:31.456 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:31.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:31.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:31.457 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:49:31.457 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:31.457 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:31.457 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:49:31.457 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:31.458 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:31.458 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:31.458 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:49:31.458 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:31.459 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:31.459 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:31.459 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:49:31.459 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:31.459 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:49:31.460 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:49:31.460 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:49:31.460 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:31.460 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:49:31.461 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.492123_183.1 2021-09-16T19:49:36.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:36.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:36.464 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:49:36.511 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:49:36.512 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:49:36.591 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:49:36.592 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:49:36.592 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:49:36.592 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:49:36.593 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:49:36.593 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:49:36.593 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:36.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:36.594 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:49:36.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:36.594 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:49:36.594 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:36.595 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:36.595 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:49:36.595 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:36.595 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:49:36.596 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:36.596 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:49:36.596 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:36.596 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:36.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:36.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:36.597 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:49:36.597 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:36.598 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:36.598 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:49:36.598 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:36.598 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:36.599 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:36.599 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:49:36.599 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:36.599 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:36.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:36.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:36.600 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:49:36.600 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:36.600 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:36.601 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.839033_212.1", 2021-09-16T19:49:36.601 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:36.601 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:36.601 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:36.601 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.839033Z", 2021-09-16T19:49:36.602 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:36.602 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:36.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:36.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:36.603 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:49:36.603 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:36.603 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:36.603 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:49:36.603 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:36.604 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:36.604 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:36.604 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:49:36.604 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:36.604 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:36.605 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:36.605 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:49:36.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:36.605 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:49:36.605 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:49:36.606 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:49:36.606 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:36.606 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:49:36.607 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.839033_212.1 2021-09-16T19:49:41.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:41.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:41.610 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:49:41.652 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:49:41.652 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:49:41.701 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:49:41.702 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:49:41.702 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:49:41.702 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:49:41.703 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:49:41.703 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:49:41.703 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:41.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:41.704 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:49:41.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:41.704 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:49:41.705 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:41.705 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:41.705 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:49:41.705 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:41.706 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:49:41.706 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:41.706 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:49:41.706 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:41.706 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:41.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:41.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:41.707 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:49:41.707 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:41.708 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:41.708 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:49:41.708 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:41.709 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:41.709 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:41.709 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:49:41.709 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:41.709 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:41.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:41.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:41.710 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:49:41.710 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:41.711 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:41.711 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:49:41.711 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:41.711 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:41.711 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:41.712 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:49:41.712 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:41.712 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:41.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:41.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:41.713 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:49:41.713 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:41.713 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:41.713 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:49:41.714 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:41.714 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:41.714 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:41.714 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:49:41.715 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:41.715 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:41.715 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:41.715 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:49:41.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:41.716 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:49:41.716 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:49:41.716 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:49:41.716 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:41.717 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:49:41.718 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.488881_181.1 2021-09-16T19:49:46.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:46.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:46.721 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:49:46.766 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:49:46.766 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:49:46.818 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:49:46.818 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:49:46.819 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:49:46.819 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:49:46.819 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:49:46.819 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:49:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:49:46.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:49:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:49:46.821 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:46.822 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:49:46.822 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:46.822 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:49:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:46.823 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:49:46.824 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:46.824 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:46.824 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:49:46.824 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:49:46.825 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:46.826 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:49:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.538756_188.1", 2021-09-16T19:49:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:46.827 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.538756Z", 2021-09-16T19:49:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:46.828 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:49:46.829 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:49:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:46.830 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:46.831 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:49:46.831 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:46.831 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:46.831 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:49:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:49:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:49:46.832 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:49:46.833 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:46.833 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:49:46.834 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.538756_188.1 2021-09-16T19:49:51.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:51.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:51.836 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:49:51.909 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:49:51.910 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:49:52.297 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:49:52.297 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:49:52.297 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:49:52.298 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:49:52.298 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:49:52.298 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:49:52.299 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:52.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:52.299 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:49:52.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:52.300 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:49:52.300 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:52.300 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:52.301 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:49:52.301 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:52.301 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:49:52.301 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:52.302 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:49:52.302 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:52.302 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:52.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:52.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:52.304 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:49:52.304 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:52.304 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:52.305 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:49:52.305 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:52.306 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:52.306 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:52.307 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:49:52.307 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:52.307 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:52.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:52.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:52.308 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:49:52.308 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:52.308 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:52.308 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.508383_184.1", 2021-09-16T19:49:52.308 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:52.309 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:52.309 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:52.309 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.508383Z", 2021-09-16T19:49:52.309 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:52.309 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:52.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:52.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:52.310 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:49:52.310 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:52.311 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:52.311 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:49:52.311 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:52.311 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:52.311 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:52.312 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:49:52.312 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:52.312 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:52.312 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:52.313 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:49:52.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:52.313 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:49:52.313 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:49:52.314 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:49:52.314 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:52.314 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:49:52.314 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.508383_184.1 2021-09-16T19:49:57.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:57.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:49:57.318 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:49:57.386 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:49:57.386 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:49:57.435 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:49:57.436 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:49:57.436 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:49:57.436 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:49:57.436 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:49:57.437 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:49:57.437 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:57.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:57.437 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:49:57.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:57.438 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:49:57.438 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:57.438 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:57.439 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:49:57.439 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:57.439 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:49:57.439 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:57.439 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:49:57.440 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:57.440 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:57.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:57.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:57.441 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:49:57.441 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:57.441 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:57.441 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:49:57.441 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:57.442 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:57.442 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:57.442 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:49:57.442 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:57.443 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:57.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:57.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:57.443 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:49:57.444 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:57.444 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:57.444 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:49:57.444 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:57.444 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:57.445 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:57.445 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:49:57.445 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:57.445 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:57.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:57.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:49:57.446 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:49:57.446 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:49:57.446 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:49:57.447 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:49:57.447 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:49:57.447 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:49:57.447 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:49:57.447 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:49:57.448 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:49:57.448 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:57.448 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:57.448 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:49:57.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:49:57.449 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:49:57.449 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:49:57.449 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:49:57.449 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:49:57.450 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:49:57.451 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:50:02.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:02.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:02.454 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:50:02.498 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:50:02.498 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:50:02.551 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:50:02.551 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:50:02.551 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:50:02.552 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:50:02.552 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:50:02.552 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:50:02.552 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:02.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:02.553 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:50:02.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:02.553 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:50:02.553 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:02.554 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:02.554 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:50:02.554 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:02.554 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:50:02.555 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:02.555 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:50:02.555 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:02.555 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:02.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:02.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:02.556 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:50:02.556 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:02.556 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:02.557 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:50:02.557 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:02.557 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:02.557 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:02.558 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:50:02.558 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:02.558 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:02.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:02.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:02.559 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:50:02.559 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:02.559 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:02.559 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.563375_192.1", 2021-09-16T19:50:02.560 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:02.560 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:02.560 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:02.560 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.563375Z", 2021-09-16T19:50:02.560 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:02.561 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:02.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:02.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:02.561 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:50:02.561 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:02.562 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:02.562 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:50:02.562 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:02.562 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:02.563 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:02.563 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:50:02.563 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:02.563 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:02.563 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:02.564 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:50:02.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:02.564 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:50:02.564 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:50:02.564 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:50:02.565 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:02.565 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:50:02.566 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.563375_192.1 2021-09-16T19:50:07.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:07.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:07.569 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:50:07.620 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:50:07.620 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:50:07.850 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:50:07.850 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:50:07.850 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:50:07.851 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:50:07.851 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:50:07.851 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:50:07.851 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:07.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:07.852 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:50:07.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:07.852 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:50:07.853 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:07.853 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:07.853 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:50:07.854 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:07.854 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:50:07.854 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:07.855 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:50:07.855 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:07.856 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:07.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:07.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:07.856 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:50:07.857 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:07.857 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:07.857 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:50:07.857 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:07.857 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:07.858 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:07.858 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:50:07.858 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:07.858 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:07.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:07.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:07.859 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:50:07.859 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:07.859 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:07.860 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:50:07.860 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:07.860 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:07.860 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:07.861 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:50:07.861 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:07.861 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:07.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:07.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:07.862 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:50:07.862 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:07.862 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:07.862 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:50:07.863 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:07.863 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:07.863 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:07.863 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:50:07.864 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:07.864 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:07.864 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:07.864 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:50:07.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:07.865 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:50:07.865 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:50:07.865 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:50:07.865 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:07.866 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:50:07.866 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:50:12.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:12.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:12.869 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:50:12.923 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:50:12.923 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:50:13.097 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:50:13.097 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:50:13.098 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:50:13.098 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:50:13.098 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:50:13.098 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:50:13.099 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:13.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:13.099 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:50:13.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:13.099 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:50:13.100 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:13.100 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:13.100 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:50:13.100 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:13.101 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:50:13.101 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:13.101 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:50:13.101 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:13.101 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:13.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:13.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:13.102 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:50:13.103 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:13.103 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:13.104 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:50:13.104 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:13.104 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:13.105 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:13.105 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:50:13.106 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:13.107 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:13.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:13.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:13.108 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:50:13.108 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:13.109 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:13.109 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.538756_188.1", 2021-09-16T19:50:13.110 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:13.110 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:13.110 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:13.111 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.538756Z", 2021-09-16T19:50:13.111 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:13.112 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:13.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:13.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:13.113 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:50:13.113 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:13.114 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:13.114 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:50:13.115 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:13.115 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:13.115 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:13.116 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:50:13.116 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:13.117 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:13.117 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:13.117 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:50:13.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:13.118 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:50:13.119 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:50:13.119 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:50:13.119 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:13.120 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:50:13.121 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.538756_188.1 2021-09-16T19:50:18.122 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:18.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:18.125 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:50:18.176 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:50:18.176 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:50:18.232 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:50:18.232 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:50:18.233 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:50:18.233 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:50:18.233 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:50:18.233 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:50:18.234 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:18.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:18.234 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:50:18.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:18.235 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:50:18.235 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:18.235 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:18.235 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:50:18.235 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:18.236 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:50:18.236 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:18.236 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:50:18.236 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:18.236 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:18.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:18.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:18.237 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:50:18.237 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:18.238 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:18.238 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:50:18.238 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:18.238 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:18.238 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:18.239 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:50:18.239 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:18.239 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:18.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:18.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:18.240 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:50:18.240 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:18.240 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:18.240 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:50:18.241 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:18.241 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:18.241 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:18.241 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:50:18.241 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:18.242 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:18.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:18.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:18.242 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:50:18.243 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:18.243 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:18.243 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:50:18.243 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:18.244 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:18.244 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:18.244 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:50:18.244 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:18.244 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:18.245 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:18.245 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:50:18.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:18.245 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:50:18.245 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:50:18.246 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:50:18.246 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:18.246 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:50:18.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.488881_181.1 2021-09-16T19:50:23.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:23.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:23.251 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:50:23.299 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:50:23.300 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:50:23.358 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:50:23.359 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:50:23.359 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:50:23.359 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:50:23.359 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:50:23.360 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:50:23.360 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:23.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:23.361 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:50:23.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:23.361 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:50:23.361 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:23.362 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:23.362 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:50:23.362 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:23.362 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:50:23.362 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:23.363 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:50:23.363 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:23.363 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:23.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:23.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:23.364 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:50:23.364 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:23.364 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:23.364 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:50:23.365 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:23.365 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:23.365 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:23.366 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:50:23.366 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:23.366 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:23.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:23.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:23.367 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:50:23.367 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:23.367 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:23.368 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:50:23.368 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:23.368 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:23.368 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:23.369 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:50:23.369 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:23.369 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:23.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:23.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:23.370 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:50:23.370 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:23.370 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:23.370 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:50:23.371 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:23.371 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:23.371 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:23.371 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:50:23.372 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:23.372 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:23.372 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:23.372 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:50:23.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:23.373 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:50:23.373 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:50:23.373 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:50:23.373 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:23.374 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:50:23.375 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:50:28.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:28.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:28.379 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:50:28.478 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:50:28.478 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:50:28.624 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:50:28.625 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:50:28.625 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:50:28.626 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:50:28.626 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:50:28.626 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:50:28.626 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:28.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:28.627 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:50:28.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:28.627 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:50:28.627 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:28.628 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:28.628 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:50:28.628 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:28.628 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:50:28.628 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:28.629 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:50:28.629 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:28.629 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:28.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:28.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:28.630 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:50:28.630 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:28.630 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:28.631 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:50:28.631 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:28.632 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:28.632 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:28.632 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:50:28.633 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:28.633 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:28.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:28.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:28.634 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:50:28.634 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:28.634 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:28.634 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:50:28.635 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:28.635 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:28.635 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:28.635 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:50:28.636 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:28.636 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:28.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:28.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:28.637 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:50:28.637 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:28.637 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:28.637 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:50:28.638 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:28.638 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:28.638 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:28.638 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:50:28.638 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:28.639 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:28.639 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:28.639 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:50:28.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:28.640 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:50:28.640 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:50:28.640 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:50:28.640 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:28.640 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:50:28.642 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821521.444824_245.1 target=1_1631821518.489753_182.1 2021-09-16T19:50:33.644 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2021-09-16T19:50:33.646 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-39 2021-09-16T19:50:33.670 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=htrfwo-40 2021-09-16T19:50:33.699 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=htrfwo-41 2021-09-16T19:50:33.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:33.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:33.738 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:50:33.783 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:50:33.783 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:50:33.836 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:50:33.851 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:50:33.851 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:50:33.851 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:50:33.851 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:50:33.852 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:50:33.852 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:50:33.852 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:50:33.852 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:50:33.853 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:50:33.853 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:50:33.853 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821833.719264_267.1", 2021-09-16T19:50:33.853 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:50:33.719264Z" 2021-09-16T19:50:33.853 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:50:33.854 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:50:33.854 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:50:33.854 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:17.633408Z" 2021-09-16T19:50:33.854 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:50:33.855 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:50:33.856 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:50:33.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:33.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:33.857 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:50:33.899 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:50:33.900 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:50:33.950 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:50:33.951 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:50:33.951 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:50:33.951 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:50:33.951 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:50:33.952 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:50:33.952 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:33.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:33.952 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:50:33.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:33.953 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:50:33.953 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:33.953 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:33.953 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:50:33.954 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:33.954 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:50:33.954 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:33.954 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:50:33.955 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:33.955 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:33.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:33.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:33.955 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:50:33.956 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:33.956 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:33.956 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:50:33.956 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:33.957 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:33.957 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:33.957 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:50:33.957 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:33.957 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:33.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:33.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:33.958 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:50:33.958 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:33.959 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:33.959 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:50:33.959 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:33.959 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:33.959 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:33.960 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:50:33.960 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:33.960 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:33.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:33.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:33.961 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:50:33.961 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:33.961 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:33.961 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:50:33.962 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:33.962 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:33.962 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:33.962 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:50:33.962 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:33.963 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:33.963 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:33.963 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:50:33.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:33.964 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:50:33.964 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:50:33.964 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:50:33.964 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:33.965 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:50:33.966 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:50:38.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:38.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:38.969 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:50:39.019 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:50:39.020 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:50:39.112 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:50:39.113 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:50:39.113 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:50:39.113 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:50:39.114 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:50:39.114 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:50:39.114 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:39.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:39.114 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:50:39.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:39.115 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:50:39.115 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:39.115 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:39.116 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:50:39.116 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:39.116 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:50:39.116 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:39.116 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:50:39.117 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:39.117 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:39.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:39.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:39.118 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:50:39.118 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:39.118 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:39.118 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:50:39.119 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:39.119 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:39.119 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:39.119 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:50:39.120 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:39.120 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:39.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:39.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:39.121 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:50:39.121 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:39.122 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:39.122 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:50:39.122 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:39.122 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:39.123 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:39.123 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:50:39.123 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:39.123 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:39.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:39.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:39.124 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:50:39.124 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:39.124 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:39.125 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:50:39.125 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:39.125 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:39.126 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:39.126 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:50:39.126 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:39.126 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:39.127 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:39.127 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:50:39.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:39.127 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:50:39.128 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:50:39.128 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:50:39.128 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:39.128 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:50:39.130 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.543574_190.1 2021-09-16T19:50:44.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:44.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:44.133 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:50:44.225 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:50:44.225 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:50:44.361 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:50:44.363 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:50:44.363 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:50:44.363 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:50:44.363 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:50:44.364 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:50:44.364 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:44.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:44.364 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:50:44.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:44.365 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:50:44.365 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:44.365 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:44.365 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:50:44.366 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:44.366 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:50:44.366 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:44.366 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:50:44.367 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:44.367 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:44.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:44.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:44.368 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:50:44.368 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:44.368 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:44.368 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:50:44.369 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:44.369 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:44.369 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:44.369 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:50:44.369 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:44.370 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:44.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:44.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:44.370 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:50:44.371 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:44.371 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:44.371 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:50:44.371 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:44.372 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:44.372 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:44.372 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:50:44.372 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:44.373 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:44.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:44.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:44.373 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:50:44.374 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:44.374 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:44.374 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:50:44.374 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:44.375 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:44.375 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:44.375 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:50:44.375 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:44.375 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:44.376 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:44.376 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:50:44.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:44.377 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:50:44.377 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:50:44.377 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:50:44.377 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:44.378 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:50:44.379 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.513492_186.1 2021-09-16T19:50:49.381 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:49.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:49.382 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:50:49.452 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:50:49.452 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:50:49.528 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:50:49.529 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:50:49.529 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:50:49.529 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:50:49.529 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:50:49.529 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:50:49.530 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:49.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:49.530 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:50:49.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:49.531 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:50:49.531 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:49.531 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:49.531 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:50:49.532 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:49.532 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:50:49.532 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:49.532 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:50:49.532 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:49.533 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:49.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:49.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:49.533 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:50:49.534 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:49.534 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:49.534 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:50:49.534 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:49.534 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:49.535 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:49.535 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:50:49.535 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:49.536 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:49.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:49.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:49.536 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:50:49.536 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:49.537 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:49.537 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:50:49.537 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:49.537 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:49.538 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:49.538 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:50:49.538 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:49.538 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:49.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:49.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:49.539 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:50:49.539 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:49.539 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:49.540 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:50:49.540 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:49.540 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:49.540 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:49.540 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:50:49.541 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:49.541 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:49.541 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:49.541 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:50:49.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:49.542 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:50:49.542 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:50:49.542 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:50:49.542 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:49.543 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:50:49.544 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.513492_186.1 2021-09-16T19:50:54.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:54.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:54.547 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:50:54.594 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:50:54.594 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:50:54.648 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:50:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:50:54.648 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:50:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:50:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:50:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:50:54.649 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:50:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:50:54.650 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:50:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:54.651 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:50:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:50:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:54.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:50:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:54.653 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:50:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:54.654 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:50:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:54.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:50:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.668185_197.1", 2021-09-16T19:50:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:54.657 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.668185Z", 2021-09-16T19:50:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:50:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:50:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:54.660 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:50:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:50:54.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:50:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:50:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:50:54.662 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:54.662 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:50:54.664 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.668185_197.1 2021-09-16T19:50:59.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:59.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:50:59.666 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:50:59.710 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:50:59.710 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:50:59.760 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:50:59.760 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:50:59.760 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:50:59.761 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:50:59.761 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:50:59.761 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:50:59.761 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:59.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:59.762 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:50:59.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:59.762 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:50:59.762 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:59.763 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:59.763 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:50:59.763 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:59.763 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:50:59.764 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:59.764 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:50:59.764 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:59.764 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:59.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:59.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:59.765 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:50:59.765 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:59.765 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:59.766 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:50:59.766 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:59.766 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:59.766 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:59.766 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:50:59.767 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:59.767 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:59.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:59.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:59.768 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:50:59.768 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:59.768 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:59.768 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:50:59.768 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:59.769 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:59.769 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:59.769 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:50:59.769 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:59.770 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:59.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:59.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:50:59.770 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:50:59.770 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:50:59.771 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:50:59.771 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:50:59.771 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:50:59.771 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:50:59.772 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:50:59.772 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:50:59.772 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:50:59.772 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:59.772 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:59.773 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:50:59.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:50:59.773 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:50:59.773 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:50:59.774 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:50:59.774 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:50:59.774 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:50:59.775 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:51:04.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:04.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:04.778 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:51:04.828 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:51:04.829 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:51:04.901 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:51:04.902 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:51:04.902 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:51:04.902 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:51:04.902 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:51:04.903 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:51:04.903 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:04.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:04.903 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:51:04.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:04.904 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:51:04.904 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:04.904 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:04.904 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:51:04.905 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:04.905 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:51:04.905 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:04.905 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:51:04.906 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:04.906 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:04.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:04.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:04.906 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:51:04.907 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:04.907 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:04.907 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:51:04.907 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:04.908 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:04.908 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:04.908 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:51:04.908 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:04.908 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:04.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:04.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:04.909 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:51:04.909 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:04.909 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:04.910 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:51:04.910 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:04.910 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:04.910 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:04.911 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:51:04.911 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:04.911 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:04.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:04.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:04.912 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:51:04.912 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:04.912 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:04.912 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:51:04.913 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:04.913 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:04.913 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:04.913 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:51:04.913 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:04.914 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:04.914 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:04.914 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:51:04.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:04.915 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:51:04.915 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:51:04.915 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:51:04.915 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:04.915 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:51:04.917 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:51:09.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:09.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:09.920 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:51:10.013 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:51:10.013 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:51:10.185 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:51:10.186 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:51:10.186 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:51:10.186 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:51:10.186 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:51:10.187 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:51:10.187 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:10.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:10.187 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:51:10.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:10.188 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:51:10.188 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:10.188 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:10.188 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:51:10.189 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:10.189 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:51:10.189 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:10.189 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:51:10.190 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:10.190 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:10.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:10.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:10.191 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:51:10.191 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:10.191 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:10.191 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:51:10.191 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:10.192 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:10.192 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:10.192 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:51:10.192 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:10.193 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:10.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:10.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:10.193 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:51:10.193 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:10.194 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:10.194 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.538756_188.1", 2021-09-16T19:51:10.194 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:10.194 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:10.195 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:10.195 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.538756Z", 2021-09-16T19:51:10.195 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:10.195 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:10.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:10.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:10.196 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:51:10.196 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:10.196 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:10.197 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:51:10.197 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:10.197 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:10.197 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:10.198 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:51:10.198 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:10.198 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:10.198 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:10.198 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:51:10.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:10.199 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:51:10.199 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:51:10.199 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:51:10.200 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:10.200 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:51:10.201 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.538756_188.1 2021-09-16T19:51:15.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:15.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:15.204 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:51:15.271 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:51:15.271 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:51:15.385 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:51:15.385 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:51:15.385 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:51:15.386 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:51:15.386 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:51:15.386 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:51:15.386 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:15.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:15.387 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:51:15.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:15.387 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:51:15.388 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:15.388 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:15.388 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:51:15.389 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:15.389 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:51:15.389 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:15.390 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:51:15.390 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:15.390 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:15.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:15.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:15.391 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:51:15.391 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:15.391 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:15.392 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:51:15.392 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:15.392 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:15.392 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:15.393 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:51:15.393 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:15.393 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:15.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:15.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:15.394 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:51:15.394 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:15.394 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:15.395 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.492123_183.1", 2021-09-16T19:51:15.395 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:15.395 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:15.396 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:15.396 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.492123Z", 2021-09-16T19:51:15.396 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:15.396 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:15.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:15.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:15.397 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:51:15.397 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:15.397 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:15.398 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:51:15.398 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:15.398 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:15.398 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:15.399 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:51:15.399 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:15.399 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:15.399 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:15.400 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:51:15.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:15.400 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:51:15.400 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:51:15.400 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:51:15.401 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:15.401 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:51:15.402 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.492123_183.1 2021-09-16T19:51:20.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:20.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:20.405 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:51:20.448 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:51:20.449 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:51:20.498 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:51:20.499 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:51:20.499 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:51:20.499 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:51:20.499 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:51:20.500 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:51:20.500 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:20.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:20.500 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:51:20.501 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:20.501 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:51:20.501 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:20.501 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:20.501 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:51:20.502 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:20.502 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:51:20.502 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:20.502 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:51:20.503 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:20.503 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:20.503 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:20.503 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:20.504 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:51:20.504 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:20.504 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:20.504 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:51:20.504 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:20.505 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:20.505 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:20.505 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:51:20.505 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:20.505 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:20.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:20.506 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:20.506 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:51:20.506 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:20.507 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:20.507 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.492123_183.1", 2021-09-16T19:51:20.507 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:20.507 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:20.508 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:20.508 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.492123Z", 2021-09-16T19:51:20.508 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:20.508 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:20.508 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:20.509 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:20.509 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:51:20.509 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:20.509 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:20.509 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:51:20.510 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:20.510 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:20.510 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:20.510 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:51:20.511 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:20.511 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:20.511 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:20.511 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:51:20.511 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:20.512 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:51:20.512 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:51:20.512 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:51:20.512 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:20.512 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:51:20.514 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.492123_183.1 2021-09-16T19:51:25.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:25.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:25.516 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:51:25.563 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:51:25.564 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:51:25.620 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:51:25.620 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:51:25.621 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:51:25.621 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:51:25.621 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:51:25.621 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:51:25.622 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:25.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:25.622 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:51:25.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:25.623 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:51:25.623 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:25.623 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:25.623 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:51:25.623 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:25.624 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:51:25.624 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:25.624 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:51:25.624 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:25.625 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:25.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:25.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:25.625 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:51:25.625 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:25.626 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:25.626 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:51:25.626 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:25.626 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:25.627 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:25.627 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:51:25.627 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:25.628 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:25.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:25.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:25.628 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:51:25.629 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:25.629 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:25.629 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:51:25.629 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:25.630 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:25.630 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:25.630 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:51:25.630 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:25.631 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:25.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:25.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:25.631 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:51:25.632 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:25.632 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:25.632 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:51:25.632 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:25.633 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:25.633 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:25.633 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:51:25.633 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:25.633 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:25.634 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:25.634 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:51:25.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:25.634 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:51:25.635 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:51:25.635 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:51:25.635 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:25.635 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:51:25.636 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.488881_181.1 2021-09-16T19:51:30.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:30.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:30.639 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:51:30.690 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:51:30.691 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:51:30.753 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:51:30.753 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:51:30.753 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:51:30.754 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:51:30.754 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:51:30.754 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:51:30.754 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:30.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:30.755 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:51:30.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:30.755 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:51:30.756 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:30.756 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:30.756 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:51:30.756 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:30.756 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:51:30.757 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:30.757 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:51:30.757 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:30.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:30.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:30.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:30.758 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:51:30.758 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:30.759 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:30.759 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:51:30.759 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:30.759 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:30.760 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:30.760 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:51:30.760 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:30.761 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:30.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:30.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:30.761 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:51:30.762 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:30.762 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:30.762 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.722236_198.1", 2021-09-16T19:51:30.762 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:30.763 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:30.763 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:30.763 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.722236Z", 2021-09-16T19:51:30.763 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:30.764 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:30.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:30.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:30.764 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:51:30.764 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:30.765 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:30.765 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:51:30.765 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:30.765 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:30.766 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:30.766 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:51:30.766 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:30.766 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:30.767 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:30.767 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:51:30.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:30.767 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:51:30.768 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:51:30.768 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:51:30.768 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:30.768 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:51:30.770 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.722236_198.1 2021-09-16T19:51:35.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:35.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:35.773 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:51:35.825 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:51:35.825 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:51:35.919 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:51:35.919 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:51:35.919 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:51:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:51:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:51:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:51:35.920 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:35.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:35.921 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:51:35.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:35.921 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:51:35.921 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:35.922 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:35.922 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:51:35.922 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:35.922 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:51:35.923 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:35.923 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:51:35.923 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:35.923 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:35.923 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:35.924 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:35.924 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:51:35.924 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:35.924 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:35.925 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:51:35.926 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:35.926 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:35.927 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:35.927 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:51:35.928 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:35.928 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:35.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:35.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:35.930 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:51:35.930 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:35.931 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.511559_185.1", 2021-09-16T19:51:35.932 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:35.933 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.511559Z", 2021-09-16T19:51:35.934 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:35.934 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:35.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:35.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:35.936 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:51:35.936 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:35.937 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:35.937 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:51:35.938 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:51:35.939 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:35.940 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:51:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:51:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:51:35.941 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:51:35.942 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:35.942 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:51:35.944 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.511559_185.1 2021-09-16T19:51:40.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:40.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:40.947 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:51:41.041 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:51:41.041 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:51:41.260 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:51:41.261 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:51:41.261 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:51:41.261 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:51:41.261 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:51:41.262 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:51:41.262 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:41.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:41.262 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:51:41.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:41.263 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:51:41.263 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:41.263 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:41.263 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:51:41.264 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:41.264 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:51:41.264 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:41.264 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:51:41.264 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:41.265 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:41.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:41.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:41.265 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:51:41.266 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:41.266 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:41.266 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:51:41.266 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:41.267 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:41.267 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:41.267 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:51:41.267 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:41.268 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:41.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:41.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:41.268 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:51:41.268 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:41.269 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:41.269 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:51:41.269 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:41.269 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:41.270 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:41.270 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:51:41.270 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:41.270 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:41.270 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:41.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:41.271 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:51:41.271 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:41.271 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:41.272 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:51:41.272 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:41.272 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:41.272 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:41.272 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:51:41.273 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:41.273 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:41.273 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:41.273 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:51:41.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:41.274 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:51:41.274 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:51:41.274 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:51:41.275 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:41.275 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:51:41.276 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.513492_186.1 2021-09-16T19:51:46.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:46.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:46.279 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:51:46.322 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:51:46.322 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:51:46.411 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:51:46.412 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:51:46.412 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:51:46.412 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:51:46.413 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:51:46.413 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:51:46.413 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:46.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:46.413 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:51:46.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:46.414 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:51:46.414 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:46.414 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:46.415 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:51:46.415 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:46.415 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:51:46.415 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:46.416 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:51:46.416 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:46.416 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:46.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:46.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:51:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:51:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:51:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:51:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:51:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:46.421 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:51:46.421 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:46.421 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:46.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:46.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:46.422 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:51:46.422 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:46.422 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:46.422 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:51:46.422 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:51:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:51:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:51:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:51:46.425 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:51:46.425 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:46.425 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:51:46.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:51:51.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:51.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:51.428 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:51:51.470 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:51:51.471 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:51:51.520 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:51:51.520 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:51:51.521 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:51:51.521 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:51:51.521 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:51:51.521 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:51:51.522 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:51.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:51.522 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:51:51.522 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:51.523 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:51:51.523 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:51.523 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:51.523 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:51:51.523 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:51.524 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:51:51.524 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:51.524 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:51:51.524 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:51.525 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:51.525 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:51.525 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:51.525 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:51:51.525 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:51.526 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:51.526 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:51:51.526 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:51.526 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:51.527 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:51.527 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:51:51.527 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:51.527 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:51.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:51.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:51.528 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:51:51.528 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:51.528 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:51.529 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:51:51.529 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:51.529 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:51.529 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:51.529 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:51:51.530 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:51.530 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:51.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:51.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:51.531 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:51:51.531 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:51.531 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:51.531 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:51:51.531 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:51.532 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:51.532 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:51.532 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:51:51.532 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:51.532 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:51.533 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:51.533 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:51:51.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:51.533 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:51:51.534 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:51:51.534 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:51:51.534 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:51.534 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:51:51.535 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:51:56.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:56.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:51:56.539 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:51:56.581 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:51:56.581 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:51:56.631 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:51:56.631 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:51:56.631 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:51:56.631 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:51:56.632 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:51:56.632 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:51:56.632 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:56.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:56.632 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:51:56.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:56.633 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:51:56.633 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:56.633 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:56.633 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:51:56.634 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:56.634 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:51:56.634 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:56.634 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:51:56.634 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:56.635 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:56.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:56.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:56.635 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:51:56.636 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:56.636 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:56.636 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:51:56.636 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:56.637 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:56.637 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:56.637 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:51:56.637 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:56.637 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:56.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:56.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:56.638 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:51:56.638 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:56.638 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:56.639 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:51:56.639 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:56.639 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:56.639 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:56.640 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:51:56.640 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:56.640 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:56.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:56.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:51:56.641 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:51:56.641 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:51:56.641 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:51:56.641 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:51:56.641 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:51:56.642 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:51:56.642 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:51:56.642 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:51:56.642 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:51:56.642 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:56.643 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:56.643 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:51:56.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:51:56.643 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:51:56.643 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:51:56.643 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:51:56.644 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:51:56.644 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:51:56.645 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.488881_181.1 2021-09-16T19:52:01.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:01.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:01.648 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:52:01.692 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:52:01.693 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:52:01.744 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:52:01.745 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:52:01.745 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:52:01.745 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:52:01.746 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:52:01.746 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:52:01.746 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:01.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:01.746 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:52:01.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:01.747 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:52:01.747 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:01.747 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:01.748 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:52:01.748 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:01.748 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:52:01.748 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:01.748 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:52:01.749 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:01.749 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:01.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:01.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:01.749 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:52:01.750 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:01.750 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:01.750 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:52:01.750 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:01.751 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:01.751 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:01.751 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:52:01.751 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:01.751 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:01.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:01.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:01.752 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:52:01.752 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:01.752 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:01.753 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.492123_183.1", 2021-09-16T19:52:01.753 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:01.753 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:01.753 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:01.753 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.492123Z", 2021-09-16T19:52:01.754 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:01.754 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:01.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:01.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:01.755 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:52:01.755 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:01.755 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:01.755 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:52:01.755 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:01.756 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:01.756 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:01.756 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:52:01.756 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:01.756 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:01.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:01.757 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:52:01.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:01.757 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:52:01.757 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:52:01.758 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:52:01.758 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:01.758 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:52:01.759 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.492123_183.1 2021-09-16T19:52:06.761 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:06.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:06.762 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:52:06.806 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:52:06.806 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:52:06.857 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:52:06.857 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:52:06.858 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:52:06.858 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:52:06.858 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:52:06.858 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:52:06.859 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:06.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:06.859 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:52:06.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:06.860 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:52:06.860 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:06.860 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:06.860 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:52:06.860 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:06.861 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:52:06.861 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:06.861 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:52:06.861 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:06.861 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:06.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:06.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:06.862 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:52:06.862 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:06.863 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:06.863 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:52:06.863 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:06.863 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:06.864 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:06.864 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:52:06.864 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:06.864 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:06.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:06.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:06.865 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:52:06.865 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:06.865 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:06.866 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.542659_189.1", 2021-09-16T19:52:06.866 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:06.866 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:06.866 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:06.866 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.542659Z", 2021-09-16T19:52:06.867 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:06.867 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:06.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:06.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:06.868 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:52:06.868 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:06.868 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:06.868 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:52:06.869 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:06.869 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:06.869 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:06.869 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:52:06.869 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:06.870 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:06.870 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:06.870 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:52:06.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:06.870 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:52:06.871 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:52:06.871 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:52:06.871 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:06.872 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:52:06.873 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.542659_189.1 2021-09-16T19:52:11.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:11.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:11.876 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:52:11.924 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:52:11.924 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:52:11.980 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:52:11.980 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:52:11.981 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:52:11.981 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:52:11.981 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:52:11.981 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:52:11.981 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:11.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:11.982 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:52:11.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:11.982 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:52:11.983 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:11.983 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:11.983 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:52:11.983 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:11.983 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:52:11.984 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:11.984 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:52:11.984 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:11.985 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:11.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:11.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:11.985 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:52:11.985 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:11.986 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:11.986 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:52:11.986 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:11.986 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:11.986 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:11.987 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:52:11.987 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:11.987 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:11.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:11.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:11.988 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:52:11.988 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:11.988 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:11.988 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:52:11.989 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:11.989 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:11.989 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:11.989 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:52:11.990 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:11.990 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:11.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:11.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:11.991 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:52:11.991 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:11.991 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:11.991 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:52:11.992 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:11.992 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:11.992 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:11.992 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:52:11.992 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:11.993 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:11.993 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:11.993 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:52:11.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:11.994 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:52:11.994 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:52:11.994 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:52:11.994 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:11.995 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:52:11.996 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.543574_190.1 2021-09-16T19:52:16.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:16.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:16.999 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:52:17.050 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:52:17.050 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:52:17.253 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:52:17.253 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:52:17.253 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:52:17.253 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:52:17.254 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:52:17.254 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:52:17.254 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:17.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:17.255 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:52:17.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:17.255 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:52:17.255 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:17.256 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:17.256 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:52:17.256 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:17.256 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:52:17.256 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:17.257 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:52:17.257 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:17.257 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:17.257 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:17.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:17.258 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:52:17.258 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:17.258 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:17.258 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:52:17.259 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:17.259 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:17.259 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:17.259 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:52:17.259 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:17.260 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:17.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:17.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:17.260 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:52:17.261 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:17.261 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:17.261 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:52:17.261 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:17.261 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:17.262 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:17.262 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:52:17.262 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:17.262 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:17.263 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:17.263 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:17.263 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:52:17.263 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:17.263 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:17.264 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:52:17.264 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:17.264 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:17.264 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:17.265 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:52:17.265 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:17.265 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:17.265 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:17.265 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:52:17.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:17.266 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:52:17.266 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:52:17.266 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:52:17.266 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:17.267 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:52:17.267 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.543574_190.1 2021-09-16T19:52:22.268 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:22.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:22.270 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:52:22.352 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:52:22.352 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:52:22.561 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:52:22.561 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:52:22.562 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:52:22.562 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:52:22.562 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:52:22.562 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:52:22.562 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:22.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:22.563 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:52:22.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:22.563 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:52:22.564 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:22.564 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:22.564 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:52:22.564 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:22.565 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:52:22.565 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:22.565 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:52:22.565 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:22.566 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:22.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:22.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:22.566 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:52:22.567 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:22.567 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:22.567 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:52:22.567 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:22.568 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:22.568 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:22.568 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:52:22.568 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:22.568 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:22.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:22.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:22.569 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:52:22.569 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:22.570 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:22.570 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:52:22.570 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:22.570 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:22.570 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:22.571 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:52:22.571 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:22.571 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:22.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:22.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:22.572 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:52:22.572 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:22.572 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:22.573 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:52:22.573 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:22.573 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:22.573 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:22.573 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:52:22.574 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:22.574 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:22.574 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:22.574 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:52:22.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:22.575 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:52:22.575 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:52:22.575 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:52:22.575 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:22.576 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:52:22.577 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:52:27.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:27.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:27.580 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:52:27.625 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:52:27.625 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:52:27.676 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:52:27.677 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:52:27.677 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:52:27.677 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:52:27.677 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:52:27.678 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:52:27.678 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:27.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:27.678 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:52:27.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:27.679 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:52:27.679 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:27.679 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:27.680 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:52:27.680 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:27.680 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:52:27.681 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:27.681 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:52:27.681 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:27.681 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:27.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:27.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:27.682 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:52:27.682 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:27.682 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:27.683 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:52:27.683 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:27.683 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:27.683 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:27.683 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:52:27.684 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:27.684 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:27.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:27.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:52:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:52:27.685 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:27.686 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:27.686 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:27.686 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:52:27.686 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:27.687 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:27.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:27.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:27.687 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:52:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:52:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:27.688 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:52:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:27.689 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:52:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:27.690 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:52:27.691 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:52:27.691 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:52:27.691 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:27.691 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:52:27.693 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.488881_181.1 2021-09-16T19:52:32.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:32.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:32.695 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:52:32.737 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:52:32.738 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:52:32.788 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:52:32.788 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:52:32.789 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:52:32.789 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:52:32.789 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:52:32.789 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:52:32.790 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:32.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:32.790 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:52:32.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:32.790 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:52:32.791 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:32.791 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:32.791 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:52:32.792 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:32.792 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:52:32.792 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:32.792 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:52:32.793 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:32.793 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:32.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:32.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:32.793 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:52:32.794 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:32.794 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:32.794 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:52:32.795 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:32.795 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:32.795 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:32.795 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:52:32.796 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:32.796 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:32.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:32.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:32.796 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:52:32.797 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:32.797 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:32.797 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.538756_188.1", 2021-09-16T19:52:32.797 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:32.797 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:32.798 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:32.798 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.538756Z", 2021-09-16T19:52:32.798 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:32.798 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:32.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:32.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:32.799 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:52:32.799 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:32.799 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:32.800 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:52:32.800 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:32.800 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:32.800 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:32.801 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:52:32.801 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:32.801 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:32.801 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:32.801 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:52:32.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:32.802 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:52:32.802 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:52:32.802 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:52:32.803 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:32.803 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:52:32.804 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.538756_188.1 2021-09-16T19:52:37.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:37.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:37.807 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:52:37.849 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:52:37.850 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:52:37.899 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:52:37.899 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:52:37.900 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:52:37.900 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:52:37.900 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:52:37.900 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:52:37.901 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:37.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:37.901 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:52:37.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:37.901 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:52:37.902 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:37.902 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:37.902 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:52:37.902 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:37.903 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:52:37.903 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:37.903 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:52:37.903 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:37.903 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:37.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:37.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:37.904 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:52:37.904 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:37.904 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:37.905 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:52:37.905 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:37.905 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:37.905 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:37.906 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:52:37.906 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:37.906 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:37.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:37.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:37.907 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:52:37.907 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:37.907 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:37.907 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.542659_189.1", 2021-09-16T19:52:37.907 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:37.908 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:37.908 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:37.908 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.542659Z", 2021-09-16T19:52:37.908 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:37.909 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:37.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:37.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:37.909 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:52:37.909 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:37.910 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:37.910 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:52:37.910 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:37.910 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:37.910 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:37.911 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:52:37.911 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:37.911 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:37.911 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:37.912 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:52:37.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:37.912 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:52:37.912 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:52:37.912 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:52:37.913 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:37.913 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:52:37.914 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.542659_189.1 2021-09-16T19:52:42.915 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:42.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:42.917 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:52:42.961 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:52:42.962 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:52:43.013 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:52:43.013 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:52:43.014 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:52:43.014 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:52:43.015 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:52:43.015 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:52:43.015 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:43.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:43.016 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:52:43.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:43.017 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:52:43.017 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:43.018 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:43.018 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:52:43.019 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:43.019 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:52:43.020 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:43.020 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:52:43.020 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:43.021 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:43.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:43.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:43.022 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:52:43.023 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:43.023 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:43.023 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:52:43.024 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:43.024 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:43.024 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:43.025 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:52:43.025 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:43.026 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:43.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:43.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:43.027 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:52:43.027 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:43.028 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:43.028 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.725649_200.1", 2021-09-16T19:52:43.029 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:43.029 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:43.029 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:43.030 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.725649Z", 2021-09-16T19:52:43.030 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:43.031 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:43.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:43.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:43.032 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:52:43.032 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:43.033 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:43.033 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:52:43.033 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:43.034 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:43.034 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:43.035 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:52:43.035 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:43.036 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:43.036 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:43.036 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:52:43.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:43.037 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:52:43.038 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:52:43.038 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:52:43.038 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:43.039 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:52:43.041 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.725649_200.1 2021-09-16T19:52:48.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:48.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:48.044 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:52:48.091 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:52:48.092 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:52:48.147 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:52:48.148 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:52:48.148 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:52:48.148 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:52:48.148 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:52:48.149 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:52:48.149 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:48.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:48.149 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:52:48.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:48.150 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:52:48.150 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:48.150 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:48.151 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:52:48.151 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:48.151 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:52:48.151 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:48.152 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:52:48.152 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:48.152 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:48.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:48.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:48.153 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:52:48.153 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:48.153 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:48.153 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:52:48.153 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:48.154 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:48.154 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:48.154 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:52:48.154 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:48.155 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:48.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:48.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:48.155 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:52:48.156 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:48.156 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:48.156 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.725649_200.1", 2021-09-16T19:52:48.156 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:48.157 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:48.157 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:48.157 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.725649Z", 2021-09-16T19:52:48.157 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:48.157 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:48.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:48.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:48.158 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:52:48.158 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:48.159 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:48.159 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:52:48.159 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:48.159 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:48.160 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:48.160 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:52:48.160 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:48.160 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:48.161 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:48.161 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:52:48.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:48.161 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:52:48.162 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:52:48.162 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:52:48.162 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:48.162 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:52:48.163 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.725649_200.1 2021-09-16T19:52:53.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:53.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:53.166 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:52:53.212 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:52:53.212 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:52:53.264 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:52:53.265 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:52:53.265 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:52:53.265 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:52:53.265 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:52:53.265 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:52:53.266 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:53.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:53.266 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:52:53.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:53.267 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:52:53.267 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:53.267 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:53.267 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:52:53.267 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:53.268 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:52:53.268 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:53.268 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:52:53.268 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:53.269 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:53.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:53.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:53.269 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:52:53.269 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:53.270 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:53.270 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:52:53.270 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:53.270 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:53.270 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:53.271 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:52:53.271 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:53.271 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:53.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:53.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:53.272 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:52:53.272 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:53.272 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:53.272 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.725649_200.1", 2021-09-16T19:52:53.273 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:53.273 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:53.273 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:53.273 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.725649Z", 2021-09-16T19:52:53.273 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:53.274 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:53.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:53.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:53.274 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:52:53.275 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:53.275 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:53.275 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:52:53.275 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:53.275 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:53.276 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:53.276 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:52:53.276 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:53.276 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:53.277 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:53.277 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:52:53.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:53.277 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:52:53.278 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:52:53.278 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:52:53.278 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:53.279 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:52:53.280 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.725649_200.1 2021-09-16T19:52:58.281 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:58.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:52:58.282 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:52:58.332 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:52:58.332 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:52:58.395 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:52:58.395 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:52:58.395 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:52:58.395 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:52:58.395 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:52:58.396 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:52:58.396 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:58.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:58.396 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:52:58.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:58.397 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:52:58.397 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:58.397 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:58.397 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:52:58.397 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:58.398 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:52:58.398 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:58.398 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:52:58.398 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:58.399 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:58.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:58.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:58.399 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:52:58.399 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:58.400 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:58.400 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:52:58.400 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:58.400 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:58.400 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:58.401 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:52:58.401 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:58.401 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:58.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:58.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:58.402 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:52:58.402 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:58.402 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:58.402 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.725649_200.1", 2021-09-16T19:52:58.403 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:58.403 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:58.403 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:58.403 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.725649Z", 2021-09-16T19:52:58.403 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:58.404 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:58.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:58.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:52:58.404 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:52:58.404 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:52:58.405 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:52:58.405 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:52:58.405 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:52:58.405 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:52:58.405 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:52:58.406 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:52:58.406 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:52:58.406 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:58.406 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:58.407 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:52:58.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:52:58.407 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:52:58.407 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:52:58.407 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:52:58.408 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:52:58.408 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:52:58.409 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.725649_200.1 2021-09-16T19:53:03.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:03.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:03.411 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:53:03.476 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:53:03.476 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:53:03.904 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:53:03.904 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:53:03.904 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:53:03.905 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:53:03.905 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:53:03.905 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:53:03.905 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:03.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:03.906 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:53:03.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:03.906 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:53:03.906 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:03.907 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:03.907 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:53:03.907 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:03.907 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:53:03.907 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:03.908 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:53:03.908 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:03.908 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:03.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:03.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:03.909 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:53:03.909 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:03.909 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:03.909 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:53:03.909 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:03.910 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:03.910 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:03.910 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:53:03.910 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:03.910 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:03.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:03.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:03.911 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:53:03.911 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:03.912 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:03.912 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:53:03.912 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:03.912 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:03.912 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:03.913 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:53:03.913 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:03.913 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:03.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:03.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:03.914 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:53:03.914 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:03.914 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:03.914 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:53:03.914 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:03.915 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:03.915 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:03.915 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:53:03.915 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:03.915 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:03.916 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:03.916 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:53:03.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:03.916 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:53:03.916 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:53:03.917 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:53:03.917 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:03.917 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:53:03.917 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:53:08.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:08.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:08.920 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:53:08.963 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:53:08.963 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:53:09.014 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:53:09.014 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:53:09.014 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:53:09.015 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:53:09.015 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:53:09.015 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:53:09.015 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:53:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:53:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:53:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:09.018 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:53:09.018 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:09.018 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:53:09.018 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:53:09.020 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:09.020 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:09.020 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:53:09.020 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:09.020 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:09.021 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:09.021 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:53:09.021 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:09.021 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:09.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:09.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:09.022 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:53:09.022 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:09.023 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:09.023 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:53:09.023 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:09.023 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:09.023 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:09.024 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:53:09.024 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:09.024 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:09.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:09.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:09.025 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:53:09.025 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:09.025 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:09.025 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:53:09.026 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:09.026 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:09.026 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:09.026 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:53:09.027 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:09.027 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:09.027 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:09.027 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:53:09.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:09.028 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:53:09.028 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:53:09.028 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:53:09.028 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:09.028 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:53:09.030 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:53:14.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:14.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:14.033 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:53:14.078 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:53:14.078 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:53:14.128 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:53:14.129 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:53:14.129 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:53:14.130 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:53:14.130 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:53:14.130 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:53:14.131 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:14.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:14.132 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:53:14.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:14.133 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:53:14.133 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:14.134 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:14.134 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:53:14.135 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:14.135 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:53:14.136 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:14.137 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:53:14.137 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:14.137 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:14.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:14.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:14.139 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:53:14.139 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:14.140 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:14.140 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:53:14.141 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:14.141 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:14.141 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:14.142 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:53:14.142 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:14.143 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:14.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:14.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:14.144 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:53:14.144 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:14.145 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:14.146 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:53:14.146 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:14.147 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:14.147 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:14.148 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:53:14.148 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:14.148 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:14.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:14.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:14.149 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:53:14.150 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:14.150 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:14.150 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:53:14.151 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:14.151 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:14.152 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:14.152 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:53:14.152 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:14.153 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:14.153 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:14.153 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:53:14.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:14.154 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:53:14.154 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:53:14.154 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:53:14.154 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:14.155 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:53:14.156 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.543574_190.1 2021-09-16T19:53:19.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:19.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:19.159 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:53:19.202 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:53:19.202 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:53:19.253 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:53:19.253 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:53:19.254 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:53:19.254 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:53:19.254 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:53:19.255 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:53:19.255 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:19.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:19.256 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:53:19.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:19.257 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:53:19.258 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:19.258 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:19.258 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:53:19.259 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:19.259 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:53:19.260 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:19.260 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:53:19.261 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:19.261 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:19.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:19.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:19.263 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:53:19.263 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:19.264 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:19.264 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:53:19.264 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:19.265 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:19.265 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:19.266 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:53:19.266 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:19.267 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:19.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:19.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:19.268 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:53:19.269 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:19.269 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:19.269 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:53:19.270 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:19.270 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:19.271 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:19.271 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:53:19.272 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:19.272 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:19.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:19.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:19.274 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:53:19.274 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:19.274 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:19.275 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:53:19.275 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:19.276 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:19.276 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:19.277 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:53:19.277 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:19.277 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:19.278 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:19.278 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:53:19.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:19.279 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:53:19.280 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:53:19.280 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:53:19.281 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:19.281 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:53:19.283 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.543574_190.1 2021-09-16T19:53:24.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:24.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:24.287 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:53:24.334 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:53:24.334 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:53:24.389 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:53:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:53:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:53:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:53:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:53:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:53:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:53:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:53:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:53:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:53:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:53:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:53:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:53:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:53:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:24.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:53:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.511559_185.1", 2021-09-16T19:53:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.511559Z", 2021-09-16T19:53:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:53:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:53:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:53:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:53:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:53:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:53:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:53:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:24.403 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:53:24.404 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.511559_185.1 2021-09-16T19:53:29.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:29.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:29.407 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:53:29.456 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:53:29.456 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:53:29.512 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:53:29.512 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:53:29.512 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:53:29.513 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:53:29.513 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:53:29.513 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:53:29.513 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:29.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:29.514 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:53:29.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:29.514 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:53:29.515 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:29.515 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:29.515 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:53:29.515 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:29.516 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:53:29.516 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:29.516 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:53:29.516 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:29.516 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:29.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:29.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:29.517 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:53:29.517 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:29.518 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:29.518 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:53:29.518 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:29.518 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:29.519 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:29.519 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:53:29.519 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:29.519 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:29.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:29.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:29.520 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:53:29.520 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:29.520 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:29.520 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:53:29.521 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:29.521 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:29.521 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:29.521 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:53:29.521 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:29.522 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:29.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:29.522 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:29.522 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:53:29.523 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:29.523 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:29.523 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:53:29.523 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:29.523 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:29.524 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:29.524 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:53:29.524 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:29.524 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:29.524 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:29.525 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:53:29.525 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:29.525 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:53:29.525 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:53:29.526 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:53:29.526 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:29.526 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:53:29.527 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.513492_186.1 2021-09-16T19:53:34.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:34.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:34.530 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:53:34.588 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:53:34.589 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:53:34.900 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:53:34.901 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:53:34.901 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:53:34.901 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:53:34.902 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:53:34.902 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:53:34.902 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:34.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:34.903 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:53:34.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:34.903 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:53:34.903 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:34.903 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:34.904 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:53:34.904 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:34.904 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:53:34.904 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:34.905 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:53:34.905 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:34.905 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:34.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:34.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:34.906 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:53:34.906 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:34.906 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:34.906 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:53:34.907 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:34.907 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:34.907 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:34.907 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:53:34.908 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:34.908 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:34.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:34.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:34.908 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:53:34.909 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:34.909 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:34.909 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.511559_185.1", 2021-09-16T19:53:34.909 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:34.909 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:34.910 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:34.910 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.511559Z", 2021-09-16T19:53:34.910 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:34.910 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:34.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:34.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:34.911 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:53:34.911 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:34.911 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:34.912 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:53:34.912 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:34.912 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:34.912 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:34.912 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:53:34.913 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:34.913 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:34.913 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:34.913 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:53:34.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:34.914 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:53:34.914 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:53:34.914 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:53:34.914 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:34.915 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:53:34.915 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.511559_185.1 2021-09-16T19:53:39.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:39.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:39.918 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:53:39.959 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:53:39.960 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:53:40.072 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:53:40.072 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:53:40.073 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:53:40.073 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:53:40.073 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:53:40.073 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:53:40.073 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:40.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:40.074 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:53:40.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:40.074 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:53:40.075 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:40.075 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:40.075 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:53:40.075 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:40.075 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:53:40.076 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:40.076 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:53:40.076 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:40.077 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:40.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:40.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:40.077 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:53:40.077 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:40.078 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:40.078 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:53:40.078 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:40.078 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:40.078 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:40.079 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:53:40.079 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:40.079 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:40.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:40.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:40.080 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:53:40.080 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:40.080 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:40.080 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:53:40.081 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:40.081 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:40.081 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:40.081 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:53:40.082 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:40.082 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:40.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:40.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:40.082 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:53:40.083 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:40.083 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:40.083 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:53:40.083 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:40.084 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:40.084 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:40.084 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:53:40.084 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:40.084 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:40.085 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:40.085 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:53:40.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:40.085 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:53:40.085 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:53:40.086 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:53:40.086 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:40.086 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:53:40.086 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:53:45.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:45.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:45.089 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:53:45.131 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:53:45.132 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:53:45.181 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:53:45.182 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:53:45.182 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:53:45.182 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:53:45.183 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:53:45.183 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:53:45.183 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:45.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:53:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:53:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:53:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:53:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:53:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:53:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:53:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:53:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:53:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.517759_187.1", 2021-09-16T19:53:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.517759Z", 2021-09-16T19:53:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:45.192 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:53:45.192 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:45.192 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:45.192 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:53:45.193 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:45.193 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:45.193 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:45.194 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:53:45.194 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:45.194 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:45.194 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:53:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:53:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:53:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:53:45.196 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:45.196 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:53:45.197 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.517759_187.1 2021-09-16T19:53:50.199 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:50.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:50.200 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:53:50.244 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:53:50.244 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:53:50.294 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:53:50.295 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:53:50.295 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:53:50.295 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:53:50.296 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:53:50.296 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:53:50.296 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:50.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:50.296 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:53:50.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:50.297 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:53:50.297 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:50.297 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:50.298 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:53:50.298 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:50.298 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:53:50.298 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:50.299 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:53:50.299 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:50.299 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:50.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:50.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:50.300 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:53:50.300 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:50.300 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:50.300 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:53:50.301 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:50.301 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:50.301 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:50.301 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:53:50.302 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:50.302 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:50.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:50.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:50.302 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:53:50.303 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:50.303 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:50.303 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.614164_194.1", 2021-09-16T19:53:50.303 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:50.303 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:50.304 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:50.304 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.614164Z", 2021-09-16T19:53:50.304 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:50.304 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:50.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:50.305 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:50.305 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:53:50.305 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:50.305 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:50.305 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:53:50.306 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:50.306 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:50.306 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:50.306 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:53:50.306 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:50.307 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:50.307 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:50.307 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:53:50.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:50.307 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:53:50.308 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:53:50.308 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:53:50.308 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:50.308 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:53:50.310 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.614164_194.1 2021-09-16T19:53:55.311 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:55.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:53:55.312 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:53:55.361 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:53:55.361 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:53:55.417 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:53:55.417 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:53:55.417 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:53:55.417 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:53:55.418 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:53:55.418 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:53:55.418 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:55.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:55.419 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:53:55.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:55.419 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:53:55.419 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:55.420 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:55.420 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:53:55.420 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:55.420 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:53:55.420 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:55.421 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:53:55.421 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:55.421 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:55.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:55.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:55.422 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:53:55.422 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:55.422 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:55.422 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:53:55.423 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:55.423 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:55.423 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:55.423 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:53:55.424 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:55.424 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:55.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:55.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:55.424 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:53:55.425 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:55.425 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:55.425 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:53:55.425 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:55.426 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:55.426 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:55.426 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:53:55.426 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:55.427 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:55.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:55.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:53:55.427 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:53:55.427 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:53:55.428 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:53:55.428 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:53:55.428 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:53:55.428 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:53:55.428 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:53:55.429 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:53:55.429 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:53:55.429 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:55.429 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:55.429 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:53:55.430 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:53:55.430 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:53:55.430 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:53:55.430 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:53:55.431 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:53:55.431 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:53:55.432 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:54:00.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:00.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:00.435 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:54:00.484 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:54:00.484 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:54:00.548 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:54:00.549 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:54:00.549 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:54:00.549 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:54:00.550 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:54:00.550 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:54:00.550 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:00.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:00.551 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:54:00.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:00.551 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:54:00.551 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:00.552 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:00.552 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:54:00.552 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:00.552 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:54:00.553 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:00.553 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:54:00.553 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:00.553 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:00.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:00.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:00.554 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:54:00.554 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:00.555 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:00.555 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:54:00.555 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:00.556 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:00.556 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:00.556 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:54:00.556 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:00.557 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:00.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:00.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:00.557 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:54:00.557 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:00.558 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:00.558 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:54:00.558 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:00.558 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:00.559 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:00.559 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:54:00.559 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:00.559 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:00.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:00.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:00.560 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:54:00.560 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:00.560 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:00.561 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:54:00.561 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:00.561 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:00.561 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:00.562 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:54:00.562 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:00.562 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:00.562 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:00.563 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:54:00.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:00.563 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:54:00.563 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:54:00.564 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:54:00.564 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:00.564 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:54:00.565 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:54:03.153 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T19:54:03.151+0000 7f2fcaf3e700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-16T19:54:03.213 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T19:54:03.212+0000 7f2fcaf3e700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-16T19:54:05.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:05.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:05.568 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:54:05.638 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:54:05.639 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:54:06.008 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:54:06.008 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:54:06.009 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:54:06.009 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:54:06.009 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:54:06.009 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:54:06.010 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:06.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:06.010 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:54:06.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:06.011 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:54:06.011 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:06.011 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:06.011 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:54:06.011 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:06.012 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:54:06.012 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:06.012 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:54:06.012 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:06.013 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:06.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:06.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:06.013 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:54:06.013 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:06.014 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:06.014 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:54:06.014 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:06.014 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:06.015 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:06.015 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:54:06.015 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:06.015 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:06.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:06.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:06.016 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:54:06.016 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:06.016 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:06.016 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:54:06.017 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:06.017 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:06.017 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:06.017 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:54:06.018 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:06.018 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:06.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:06.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:06.018 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:54:06.019 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:06.019 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:06.019 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:54:06.019 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:06.019 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:06.020 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:06.020 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:54:06.020 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:06.020 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:06.020 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:06.021 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:54:06.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:06.021 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:54:06.021 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:54:06.021 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:54:06.022 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:06.022 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:54:06.023 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.543574_190.1 2021-09-16T19:54:11.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:11.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:11.026 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:54:11.068 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:54:11.068 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:54:11.121 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:54:11.121 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:54:11.121 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:54:11.122 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:54:11.122 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:54:11.122 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:54:11.122 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:11.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:11.123 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:54:11.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:11.123 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:54:11.123 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:11.124 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:11.124 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:54:11.124 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:11.124 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:54:11.125 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:11.125 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:54:11.125 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:11.125 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:11.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:11.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:11.126 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:54:11.126 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:11.126 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:11.127 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:54:11.127 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:11.127 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:11.127 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:11.128 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:54:11.128 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:11.128 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:11.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:11.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:11.129 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:54:11.129 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:11.129 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:11.129 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.665525_196.1", 2021-09-16T19:54:11.130 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:11.130 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:11.130 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:11.130 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.665525Z", 2021-09-16T19:54:11.131 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:11.131 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:11.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:11.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:11.131 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:54:11.132 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:11.132 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:11.132 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:54:11.132 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:11.133 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:11.133 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:11.133 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:54:11.133 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:11.134 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:11.134 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:11.134 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:54:11.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:11.134 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:54:11.135 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:54:11.135 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:54:11.135 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:11.135 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:54:11.137 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.665525_196.1 2021-09-16T19:54:16.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:16.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:16.139 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:54:16.183 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:54:16.184 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:54:16.270 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:54:16.270 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:54:16.270 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:54:16.270 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:54:16.271 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:54:16.271 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:54:16.271 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:16.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:16.272 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:54:16.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:16.272 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:54:16.272 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:16.273 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:16.273 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:54:16.273 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:16.273 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:54:16.273 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:16.274 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:54:16.274 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:16.274 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:16.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:16.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:16.275 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:54:16.275 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:16.275 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:16.275 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:54:16.276 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:16.276 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:16.276 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:16.276 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:54:16.276 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:16.277 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:16.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:16.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:16.277 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:54:16.277 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:16.278 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:16.278 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:54:16.278 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:16.278 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:16.278 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:16.279 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:54:16.279 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:16.279 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:16.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:16.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:16.280 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:54:16.280 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:16.280 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:16.280 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:54:16.280 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:16.281 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:16.281 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:16.281 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:54:16.281 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:16.281 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:16.282 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:16.282 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:54:16.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:16.282 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:54:16.283 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:54:16.283 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:54:16.283 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:16.283 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:54:16.284 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.513492_186.1 2021-09-16T19:54:21.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:21.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:21.287 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:54:21.338 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:54:21.339 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:54:21.388 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:54:21.389 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:54:21.389 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:54:21.389 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:54:21.389 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:54:21.390 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:54:21.390 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:21.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:21.390 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:54:21.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:21.391 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:54:21.391 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:21.391 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:21.391 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:54:21.392 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:21.392 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:54:21.392 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:21.392 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:54:21.393 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:21.393 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:21.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:21.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:21.394 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:54:21.394 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:21.394 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:21.394 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:54:21.395 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:21.395 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:21.395 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:21.395 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:54:21.395 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:21.396 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:21.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:21.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:21.396 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:54:21.396 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:21.397 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:21.397 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:54:21.397 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:21.397 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:21.398 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:21.398 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:54:21.398 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:21.398 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:21.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:21.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:21.399 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:54:21.399 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:21.399 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:21.399 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:54:21.400 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:21.400 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:21.400 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:21.400 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:54:21.401 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:21.401 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:21.401 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:21.401 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:54:21.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:21.402 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:54:21.402 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:54:21.403 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:54:21.403 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:21.403 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:54:21.404 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.488881_181.1 2021-09-16T19:54:26.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:26.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:26.407 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:54:26.449 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:54:26.450 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:54:26.500 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:54:26.501 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:54:26.501 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:54:26.501 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:54:26.501 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:54:26.502 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:54:26.502 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:26.502 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:26.502 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:54:26.502 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:26.503 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:54:26.503 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:26.503 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:26.503 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:54:26.504 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:26.504 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:54:26.504 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:26.504 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:54:26.504 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:26.505 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:26.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:26.505 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:26.505 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:54:26.505 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:26.506 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:26.506 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:54:26.506 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:26.506 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:26.506 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:26.507 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:54:26.507 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:26.507 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:26.507 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:26.508 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:26.508 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:54:26.508 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:26.508 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:26.508 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:54:26.509 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:26.509 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:26.509 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:26.509 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:54:26.510 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:26.510 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:26.510 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:26.510 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:26.510 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:54:26.511 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:26.511 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:26.511 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:54:26.511 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:26.512 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:26.512 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:26.512 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:54:26.512 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:26.512 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:26.513 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:26.513 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:54:26.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:26.513 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:54:26.513 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:54:26.514 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:54:26.514 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:26.514 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:54:26.515 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.488881_181.1 2021-09-16T19:54:31.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:31.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:31.518 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:54:31.568 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:54:31.568 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:54:31.624 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:54:31.625 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:54:31.625 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:54:31.625 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:54:31.625 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:54:31.626 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:54:31.626 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:31.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:31.626 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:54:31.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:31.627 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:54:31.627 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:31.627 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:31.628 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:54:31.628 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:31.628 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:54:31.628 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:31.629 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:54:31.629 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:31.629 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:31.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:31.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:31.630 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:54:31.630 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:31.630 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:31.631 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:54:31.631 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:31.631 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:31.631 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:31.631 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:54:31.632 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:31.632 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:31.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:31.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:31.633 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:54:31.633 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:31.633 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:31.633 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:54:31.634 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:31.634 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:31.634 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:31.634 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:54:31.635 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:31.635 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:31.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:31.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:31.636 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:54:31.636 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:31.636 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:31.636 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:54:31.636 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:31.637 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:31.637 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:31.637 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:54:31.637 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:31.638 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:31.638 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:31.638 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:54:31.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:31.639 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:54:31.639 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:54:31.639 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:54:31.639 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:31.640 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:54:31.641 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:54:36.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:36.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:36.644 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:54:36.693 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:54:36.693 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:54:36.750 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:54:36.751 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:54:36.751 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:54:36.751 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:54:36.751 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:54:36.752 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:54:36.752 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:36.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:36.752 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:54:36.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:36.753 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:54:36.753 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:36.753 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:36.753 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:54:36.754 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:36.754 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:54:36.754 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:36.754 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:54:36.755 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:36.755 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:36.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:36.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:36.755 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:54:36.756 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:36.756 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:36.756 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:54:36.757 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:36.757 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:36.757 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:36.757 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:54:36.758 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:36.758 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:36.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:36.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:36.758 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:54:36.759 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:36.759 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:36.759 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821833.719264_267.1", 2021-09-16T19:54:36.759 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:36.759 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:36.760 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:36.760 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:50:33.719264Z", 2021-09-16T19:54:36.760 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:36.760 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:36.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:36.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:54:36.761 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:54:36.761 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:54:36.761 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:54:36.762 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:54:36.762 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:54:36.762 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:54:36.762 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:54:36.763 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:54:36.763 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:54:36.763 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:36.763 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:36.764 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:54:36.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:54:36.764 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:54:36.764 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:54:36.764 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:54:36.765 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:54:36.765 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:54:36.766 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:54:36.767 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:54:36.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:36.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:36.768 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:54:36.817 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:54:36.817 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:54:36.874 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:54:36.875 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:54:36.875 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:54:36.875 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:54:36.876 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:54:36.876 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:54:36.876 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:36.876 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:36.876 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:54:36.877 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:36.877 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:54:36.877 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:36.877 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:36.877 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:54:36.878 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:36.878 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:54:36.878 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:36.878 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:54:36.879 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:36.879 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:36.879 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:36.879 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:36.879 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:54:36.880 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:36.880 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:36.880 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:54:36.880 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:36.881 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:54:36.881 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:36.881 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:54:36.881 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:36.881 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:36.882 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:36.882 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:36.882 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:54:36.882 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:36.882 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:36.883 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:54:36.883 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:36.883 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:54:36.883 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:36.884 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:54:36.884 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:36.884 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:36.884 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:36.884 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:36.885 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:54:36.885 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:36.885 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:36.885 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:54:36.885 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:36.886 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:54:36.886 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:36.886 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:54:36.886 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:36.886 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:36.887 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:36.887 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:54:36.887 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:36.887 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:54:36.887 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:54:36.888 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:54:36.888 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:36.888 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:54:36.889 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.543574_190.1 2021-09-16T19:54:41.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:41.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:41.892 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:54:41.953 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:54:41.954 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:54:42.104 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:54:42.105 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:54:42.105 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:54:42.105 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:54:42.105 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:54:42.105 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:54:42.106 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:42.106 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:42.106 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:54:42.107 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:42.107 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:54:42.107 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:42.107 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:42.107 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:54:42.108 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:42.108 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:54:42.108 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:42.108 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:54:42.108 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:42.109 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:42.109 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:42.109 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:42.109 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:54:42.110 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:42.110 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:42.110 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:54:42.110 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:42.111 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:54:42.111 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:42.111 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:54:42.111 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:42.111 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:42.112 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:42.112 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:42.112 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:54:42.112 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:42.113 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:42.113 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.738934_202.1", 2021-09-16T19:54:42.113 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:42.113 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:54:42.113 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:42.114 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.738934Z", 2021-09-16T19:54:42.114 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:42.114 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:42.114 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:42.115 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:42.115 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:54:42.115 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:42.115 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:42.116 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:54:42.116 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:42.116 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:54:42.116 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:42.116 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:54:42.117 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:42.117 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:42.117 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:42.117 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:54:42.117 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:42.118 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:54:42.118 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:54:42.118 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:54:42.118 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:42.119 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:54:42.120 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.738934_202.1 2021-09-16T19:54:47.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:47.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:47.124 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:54:47.204 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:54:47.205 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:54:47.316 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:54:47.317 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:54:47.317 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:54:47.317 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:54:47.317 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:54:47.317 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:54:47.318 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:47.318 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:47.318 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:54:47.319 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:47.319 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:54:47.319 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:47.319 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:47.319 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:54:47.320 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:47.320 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:54:47.320 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:47.320 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:54:47.320 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:47.321 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:47.321 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:47.321 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:47.321 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:54:47.321 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:47.322 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:47.322 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:54:47.322 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:47.323 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:54:47.323 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:47.323 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:54:47.323 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:47.324 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:47.324 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:47.324 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:47.324 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:54:47.324 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:47.325 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:47.325 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:54:47.325 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:47.325 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:54:47.326 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:47.326 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:54:47.326 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:47.326 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:47.326 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:47.327 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:47.327 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:54:47.327 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:47.328 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:47.328 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:54:47.328 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:47.328 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:54:47.328 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:47.329 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:54:47.329 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:47.329 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:47.329 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:47.329 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:54:47.330 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:47.330 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:54:47.330 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:54:47.330 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:54:47.331 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:47.331 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:54:47.332 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.543574_190.1 2021-09-16T19:54:52.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:52.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:52.335 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:54:52.387 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:54:52.387 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:54:52.541 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:54:52.541 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:54:52.541 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:54:52.542 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:54:52.542 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:54:52.542 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:54:52.542 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:52.543 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:52.543 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:54:52.543 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:52.543 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:54:52.544 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:52.544 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:52.544 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:54:52.544 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:52.544 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:54:52.545 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:52.545 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:54:52.545 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:52.545 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:52.546 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:52.546 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:52.546 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:54:52.546 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:52.546 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:52.547 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:54:52.547 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:52.547 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:54:52.547 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:52.548 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:54:52.548 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:52.548 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:52.548 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:52.548 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:52.549 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:54:52.549 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:52.549 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:52.549 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:54:52.550 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:52.550 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:54:52.550 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:52.550 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:54:52.550 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:52.551 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:52.551 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:52.551 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:52.551 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:54:52.552 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:52.552 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:52.552 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:54:52.552 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:52.552 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:54:52.553 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:52.553 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:54:52.553 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:52.553 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:52.553 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:52.554 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:54:52.554 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:52.554 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:54:52.554 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:54:52.554 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:54:52.555 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:52.555 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:54:52.556 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.513492_186.1 2021-09-16T19:54:57.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:57.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:54:57.559 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:54:57.600 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:54:57.601 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:54:57.656 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:54:57.656 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:54:57.656 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:54:57.657 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:54:57.657 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:54:57.657 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:54:57.657 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:57.658 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:57.658 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:54:57.658 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:57.658 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:54:57.659 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:57.659 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:57.659 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:54:57.659 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:57.659 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:54:57.660 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:57.660 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:54:57.660 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:57.660 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:57.661 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:57.661 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:57.661 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:54:57.661 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:57.661 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:57.662 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:54:57.662 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:57.662 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:54:57.662 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:57.663 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:54:57.663 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:57.663 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:57.663 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:57.663 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:57.664 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:54:57.664 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:57.664 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:57.664 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.517759_187.1", 2021-09-16T19:54:57.664 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:57.665 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:54:57.665 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:57.665 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.517759Z", 2021-09-16T19:54:57.665 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:57.666 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:57.666 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:57.666 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:54:57.666 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:54:57.667 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:54:57.667 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:54:57.667 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:54:57.667 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:54:57.667 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:54:57.668 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:54:57.668 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:54:57.668 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:54:57.668 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:57.669 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:57.669 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:54:57.669 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:54:57.669 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:54:57.669 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:54:57.670 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:54:57.670 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:54:57.670 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:54:57.671 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.517759_187.1 2021-09-16T19:55:02.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:02.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:02.674 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:55:02.719 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:55:02.719 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:55:02.776 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:55:02.776 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:55:02.776 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:55:02.777 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:55:02.777 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:55:02.777 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:55:02.777 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:02.777 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:02.778 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:55:02.778 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:02.778 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:55:02.778 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:02.778 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:02.779 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:55:02.779 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:02.779 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:55:02.779 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:02.780 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:55:02.780 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:02.780 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:02.780 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:02.780 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:02.781 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:55:02.781 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:02.781 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:02.781 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:55:02.781 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:02.782 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:02.782 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:02.782 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:55:02.782 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:02.783 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:02.783 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:02.783 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:02.783 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:55:02.783 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:02.784 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:02.784 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:55:02.784 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:02.784 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:02.785 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:02.785 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:55:02.785 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:02.785 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:02.786 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:02.786 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:02.786 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:55:02.786 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:02.786 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:02.787 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:55:02.787 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:02.787 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:02.787 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:02.788 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:55:02.788 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:02.788 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:02.788 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:02.789 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:55:02.789 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:02.789 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:55:02.789 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:55:02.790 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:55:02.790 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:02.790 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:55:02.791 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.513492_186.1 2021-09-16T19:55:07.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:07.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:07.794 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:55:07.838 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:55:07.838 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:55:07.889 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:55:07.889 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:55:07.890 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:55:07.890 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:55:07.890 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:55:07.890 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:55:07.890 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:07.891 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:07.891 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:55:07.891 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:07.891 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:55:07.891 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:07.892 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:07.892 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:55:07.892 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:07.892 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:55:07.892 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:07.893 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:55:07.893 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:07.893 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:07.894 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:07.894 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:07.894 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:55:07.894 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:07.895 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:07.895 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:55:07.895 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:07.895 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:07.895 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:07.896 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:55:07.896 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:07.896 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:07.896 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:07.897 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:07.897 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:55:07.897 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:07.897 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:07.897 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:55:07.898 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:07.898 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:07.898 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:07.898 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:55:07.899 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:07.899 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:07.899 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:07.899 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:07.899 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:55:07.900 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:07.900 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:07.900 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:55:07.900 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:07.901 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:07.901 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:07.901 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:55:07.901 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:07.901 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:07.902 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:07.902 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:55:07.902 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:07.902 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:55:07.902 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:55:07.903 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:55:07.903 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:07.903 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:55:07.904 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.488881_181.1 2021-09-16T19:55:12.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:12.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:12.907 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:55:13.048 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:55:13.048 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:55:13.120 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:55:13.120 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:55:13.121 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:55:13.121 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:55:13.121 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:55:13.121 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:55:13.122 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:13.122 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:13.122 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:55:13.122 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:13.123 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:55:13.123 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:13.123 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:13.123 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:55:13.124 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:13.124 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:55:13.124 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:13.124 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:55:13.124 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:13.125 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:13.125 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:13.125 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:13.125 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:55:13.126 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:13.126 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:13.126 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:55:13.126 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:13.126 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:13.127 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:13.127 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:55:13.127 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:13.127 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:13.128 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:13.128 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:13.128 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:55:13.128 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:13.128 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:13.129 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.542659_189.1", 2021-09-16T19:55:13.129 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:13.129 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:13.129 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:13.130 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.542659Z", 2021-09-16T19:55:13.130 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:13.130 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:13.130 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:13.130 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:13.131 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:55:13.131 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:13.131 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:13.131 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:55:13.132 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:13.132 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:13.132 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:13.132 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:55:13.132 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:13.133 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:13.133 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:13.133 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:55:13.133 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:13.134 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:55:13.134 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:55:13.134 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:55:13.134 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:13.134 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:55:13.136 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.542659_189.1 2021-09-16T19:55:18.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:18.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:18.139 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:55:18.343 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:55:18.344 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:55:18.431 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:55:18.431 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:55:18.431 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:55:18.432 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:55:18.432 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:55:18.433 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:55:18.433 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:18.433 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:18.433 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:55:18.433 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:18.434 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:55:18.434 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:18.434 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:18.434 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:55:18.435 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:18.435 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:55:18.435 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:18.435 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:55:18.436 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:18.436 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:18.436 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:18.436 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:18.436 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:55:18.437 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:18.437 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:18.437 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:55:18.437 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:18.438 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:18.438 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:18.438 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:55:18.438 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:18.438 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:18.439 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:18.439 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:18.439 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:55:18.439 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:18.439 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:18.440 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:55:18.440 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:18.440 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:18.440 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:18.441 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:55:18.441 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:18.441 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:18.441 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:18.441 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:18.442 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:55:18.442 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:18.442 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:18.443 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:55:18.443 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:18.443 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:18.443 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:18.443 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:55:18.444 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:18.444 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:18.444 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:18.444 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:55:18.445 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:18.445 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:55:18.445 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:55:18.445 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:55:18.445 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:18.446 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:55:18.447 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.488881_181.1 2021-09-16T19:55:23.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:23.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:23.449 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:55:23.501 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:55:23.501 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:55:23.783 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:55:23.783 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:55:23.784 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:55:23.784 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:55:23.784 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:55:23.784 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:55:23.785 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:23.785 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:23.785 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:55:23.785 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:23.785 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:55:23.786 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:23.786 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:23.786 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:55:23.786 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:23.787 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:55:23.787 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:23.787 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:55:23.787 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:23.788 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:23.788 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:23.788 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:23.788 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:55:23.788 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:23.789 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:23.789 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:55:23.789 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:23.789 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:23.789 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:23.790 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:55:23.790 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:23.790 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:23.790 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:23.791 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:23.791 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:55:23.791 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:23.791 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:23.791 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.538756_188.1", 2021-09-16T19:55:23.792 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:23.792 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:23.792 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:23.792 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.538756Z", 2021-09-16T19:55:23.792 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:23.793 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:23.793 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:23.793 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:23.793 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:55:23.793 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:23.794 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:23.794 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:55:23.794 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:23.794 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:23.795 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:23.795 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:55:23.795 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:23.795 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:23.795 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:23.796 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:55:23.796 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:23.796 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:55:23.796 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:55:23.796 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:55:23.797 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:23.797 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:55:23.797 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.538756_188.1 2021-09-16T19:55:28.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:28.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:28.801 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:55:28.979 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:55:28.980 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:55:29.071 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:55:29.071 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:55:29.072 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:55:29.072 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:55:29.072 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:55:29.072 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:55:29.073 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:29.073 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:29.073 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:55:29.073 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:29.073 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:55:29.074 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:29.074 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:29.074 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:55:29.074 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:29.075 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:55:29.075 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:29.075 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:55:29.075 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:29.076 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:29.076 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:29.076 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:29.076 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:55:29.077 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:29.077 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:29.077 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:55:29.077 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:29.077 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:29.078 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:29.078 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:55:29.078 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:29.078 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:29.079 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:29.079 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:29.079 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:55:29.079 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:29.079 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:29.080 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.764797_207.1", 2021-09-16T19:55:29.080 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:29.080 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:29.080 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:29.080 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.764797Z", 2021-09-16T19:55:29.081 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:29.081 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:29.081 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:29.081 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:29.081 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:55:29.082 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:29.082 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:29.082 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:55:29.082 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:29.082 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:29.083 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:29.083 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:55:29.083 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:29.083 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:29.084 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:29.084 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:55:29.084 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:29.084 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:55:29.084 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:55:29.085 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:55:29.085 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:29.085 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:55:29.086 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.764797_207.1 2021-09-16T19:55:34.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:34.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:34.088 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:55:34.157 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:55:34.158 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:55:34.478 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:55:34.479 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:55:34.479 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:55:34.479 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:55:34.479 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:55:34.480 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:55:34.480 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:34.480 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:34.480 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:55:34.480 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:34.481 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:55:34.481 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:34.481 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:34.481 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:55:34.482 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:34.482 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:55:34.482 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:34.482 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:55:34.483 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:34.483 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:34.483 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:34.483 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:34.483 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:55:34.484 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:34.484 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:34.484 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:55:34.484 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:34.484 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:34.485 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:34.485 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:55:34.485 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:34.485 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:34.486 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:34.486 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:34.486 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:55:34.486 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:34.486 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:34.487 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.492123_183.1", 2021-09-16T19:55:34.487 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:34.487 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:34.487 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:34.488 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.492123Z", 2021-09-16T19:55:34.488 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:34.488 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:34.488 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:34.489 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:34.489 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:55:34.489 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:34.489 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:34.489 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:55:34.490 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:34.490 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:34.490 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:34.490 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:55:34.490 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:34.491 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:34.491 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:34.491 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:55:34.491 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:34.492 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:55:34.492 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:55:34.492 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:55:34.492 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:34.492 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:55:34.493 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.492123_183.1 2021-09-16T19:55:38.042 INFO:tasks.rgw.c1.client.1.smithi109.stdout:2021-09-16T19:55:38.041+0000 7f2506e7c700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-16T19:55:39.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:39.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:39.495 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:55:39.541 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:55:39.541 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:55:39.600 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:55:39.600 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:55:39.601 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:55:39.601 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:55:39.601 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:55:39.601 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:55:39.602 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:39.602 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:39.602 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:55:39.602 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:39.603 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:55:39.603 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:39.603 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:39.603 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:55:39.603 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:39.604 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:55:39.604 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:39.604 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:55:39.604 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:39.605 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:39.605 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:39.605 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:39.605 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:55:39.605 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:39.606 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:39.606 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:55:39.606 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:39.606 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:39.606 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:39.607 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:55:39.607 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:39.607 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:39.607 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:39.608 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:39.608 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:55:39.608 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:39.608 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:39.608 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:55:39.609 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:39.609 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:39.609 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:39.609 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:55:39.609 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:39.610 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:39.610 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:39.610 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:39.610 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:55:39.610 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:39.611 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:39.611 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:55:39.611 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:39.611 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:39.612 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:39.612 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:55:39.612 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:39.612 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:39.612 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:39.613 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:55:39.613 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:39.613 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:55:39.613 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:55:39.613 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:55:39.614 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:39.614 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:55:39.615 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.513492_186.1 2021-09-16T19:55:44.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:44.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:44.619 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:55:44.669 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:55:44.669 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:55:44.735 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:55:44.736 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:55:44.736 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:55:44.736 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:55:44.736 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:55:44.737 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:55:44.737 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:44.737 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:44.737 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:55:44.738 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:44.738 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:55:44.738 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:44.738 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:44.738 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:55:44.739 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:44.739 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:55:44.739 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:44.739 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:55:44.740 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:44.740 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:44.740 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:44.740 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:44.740 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:55:44.741 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:44.741 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:44.741 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:55:44.741 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:44.742 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:44.742 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:44.742 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:55:44.742 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:44.742 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:44.743 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:44.743 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:44.743 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:55:44.743 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:44.744 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:44.744 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:55:44.744 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:44.744 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:44.744 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:44.745 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:55:44.745 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:44.745 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:44.745 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:44.746 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:44.746 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:55:44.746 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:44.746 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:44.746 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:55:44.747 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:44.747 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:44.747 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:44.747 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:55:44.748 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:44.748 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:44.748 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:44.748 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:55:44.748 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:44.749 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:55:44.749 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:55:44.749 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:55:44.749 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:44.749 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:55:44.751 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.543574_190.1 2021-09-16T19:55:49.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:49.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:49.754 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:55:49.800 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:55:49.801 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:55:49.865 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:55:49.865 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:55:49.865 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:55:49.865 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:55:49.866 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:55:49.866 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:55:49.866 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:49.866 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:49.867 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:55:49.867 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:49.867 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:55:49.867 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:49.868 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:49.868 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:55:49.868 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:49.868 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:55:49.868 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:49.869 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:55:49.869 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:49.869 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:49.869 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:49.869 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:49.870 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:55:49.870 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:49.870 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:49.870 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:55:49.871 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:49.871 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:49.871 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:49.871 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:55:49.871 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:49.872 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:49.872 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:49.872 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:49.872 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:55:49.872 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:49.873 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:49.873 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:55:49.873 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:49.873 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:49.874 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:49.874 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:55:49.874 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:49.874 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:49.874 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:49.875 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:49.875 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:55:49.875 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:49.875 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:49.876 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:55:49.876 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:49.876 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:49.876 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:49.876 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:55:49.877 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:49.877 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:49.877 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:49.877 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:55:49.878 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:49.878 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:55:49.878 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:55:49.878 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:55:49.878 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:49.879 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:55:49.880 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.513492_186.1 2021-09-16T19:55:54.881 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:54.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:54.883 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:55:54.925 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:55:54.926 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:55:54.977 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:55:54.977 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:55:54.977 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:55:54.978 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:55:54.978 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:55:54.978 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:55:54.978 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:54.979 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:54.979 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:55:54.979 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:54.979 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:55:54.980 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:54.980 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:54.980 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:55:54.980 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:54.980 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:55:54.981 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:54.981 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:55:54.981 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:54.981 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:54.982 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:54.982 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:54.982 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:55:54.982 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:54.982 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:54.983 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:55:54.983 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:54.983 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:54.983 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:54.984 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:55:54.984 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:54.984 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:54.984 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:54.984 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:54.985 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:55:54.985 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:54.985 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:54.985 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.566120_193.1", 2021-09-16T19:55:54.986 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:54.986 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:54.986 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:54.986 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.566120Z", 2021-09-16T19:55:54.986 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:54.987 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:54.987 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:54.987 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:55:54.987 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:55:54.988 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:55:54.988 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:55:54.988 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:55:54.988 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:55:54.988 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:55:54.989 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:55:54.989 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:55:54.989 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:55:54.989 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:54.989 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:54.990 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:55:54.990 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:55:54.990 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:55:54.990 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:55:54.991 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:55:54.991 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:55:54.991 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:55:54.992 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.566120_193.1 2021-09-16T19:55:59.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:59.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:55:59.995 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:56:00.042 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:56:00.042 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:56:00.097 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:56:00.097 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:56:00.097 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:56:00.098 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:56:00.098 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:56:00.098 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:56:00.098 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:00.099 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:00.099 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:56:00.099 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:00.099 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:56:00.099 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:00.100 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:00.100 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:56:00.100 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:00.100 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:56:00.100 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:00.101 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:56:00.101 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:00.101 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:00.101 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:00.102 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:00.102 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:56:00.102 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:00.102 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:00.102 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:56:00.103 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:00.103 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:00.103 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:00.103 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:56:00.103 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:00.104 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:00.104 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:00.104 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:00.104 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:56:00.105 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:00.105 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:00.105 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.727919_201.1", 2021-09-16T19:56:00.105 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:00.105 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:00.106 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:00.106 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.727919Z", 2021-09-16T19:56:00.106 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:00.106 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:00.106 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:00.107 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:00.107 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:56:00.107 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:00.107 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:00.107 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:56:00.108 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:00.108 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:00.108 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:00.108 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:56:00.108 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:00.109 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:00.109 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:00.109 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:56:00.109 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:00.109 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:56:00.110 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:56:00.110 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:56:00.110 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:00.110 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:56:00.111 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.727919_201.1 2021-09-16T19:56:05.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:05.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:05.115 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:56:05.166 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:56:05.166 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:56:05.225 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:56:05.226 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:56:05.226 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:56:05.226 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:56:05.226 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:56:05.227 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:56:05.227 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:05.227 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:05.227 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:56:05.228 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:05.228 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:56:05.228 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:05.228 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:05.229 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:56:05.229 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:05.229 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:56:05.229 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:05.230 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:56:05.230 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:05.230 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:05.230 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:05.230 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:05.231 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:56:05.231 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:05.231 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:05.231 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:56:05.232 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:05.232 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:05.232 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:05.232 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:56:05.233 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:05.233 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:05.233 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:05.233 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:05.233 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:56:05.234 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:05.234 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:05.234 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:56:05.234 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:05.235 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:05.235 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:05.235 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:56:05.235 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:05.235 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:05.236 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:05.236 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:05.236 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:56:05.236 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:05.236 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:05.237 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:56:05.237 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:05.237 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:05.237 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:05.238 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:56:05.238 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:05.239 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:05.239 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:05.239 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:56:05.240 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:05.240 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:56:05.241 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:56:05.241 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:56:05.241 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:05.242 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:56:05.243 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.513492_186.1 2021-09-16T19:56:10.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:10.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:10.246 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:56:10.292 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:56:10.292 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:56:10.412 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:56:10.412 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:56:10.413 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:56:10.413 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:56:10.413 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:56:10.413 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:56:10.414 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:10.414 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:10.414 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:56:10.414 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:10.415 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:56:10.415 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:10.415 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:10.415 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:56:10.416 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:10.416 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:56:10.416 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:10.416 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:56:10.417 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:10.417 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:10.417 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:10.417 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:10.418 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:56:10.418 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:10.418 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:10.418 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:56:10.418 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:10.419 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:10.419 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:10.419 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:56:10.419 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:10.420 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:10.420 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:10.420 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:10.420 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:56:10.421 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:10.421 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:10.421 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:56:10.422 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:10.422 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:10.422 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:10.422 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:56:10.422 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:10.423 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:10.423 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:10.423 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:10.423 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:56:10.424 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:10.424 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:10.424 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:56:10.424 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:10.424 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:10.425 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:10.425 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:56:10.425 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:10.425 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:10.426 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:10.426 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:56:10.426 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:10.426 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:56:10.426 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:56:10.427 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:56:10.427 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:10.427 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:56:10.428 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.513492_186.1 2021-09-16T19:56:15.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:15.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:15.430 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:56:15.507 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:56:15.507 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:56:15.876 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:56:15.876 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:56:15.876 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:56:15.876 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:56:15.877 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:56:15.877 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:56:15.877 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:15.877 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:15.878 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:56:15.878 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:15.878 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:56:15.878 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:15.878 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:15.879 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:56:15.879 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:15.879 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:56:15.879 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:15.880 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:56:15.880 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:15.880 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:15.880 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:15.880 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:15.881 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:56:15.881 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:15.881 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:15.881 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:56:15.882 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:15.882 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:15.882 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:15.882 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:56:15.882 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:15.883 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:15.883 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:15.883 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:15.883 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:56:15.884 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:15.884 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:15.884 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:56:15.884 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:15.884 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:15.885 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:15.885 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:56:15.885 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:15.885 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:15.886 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:15.886 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:15.886 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:56:15.886 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:15.886 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:15.887 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:56:15.887 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:15.887 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:15.887 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:15.888 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:56:15.888 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:15.888 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:15.888 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:15.889 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:56:15.889 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:15.889 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:56:15.889 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:56:15.889 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:56:15.890 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:15.890 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:56:15.891 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.543574_190.1 2021-09-16T19:56:18.027 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2021-09-16T19:56:18.028+0000 7f0df1560700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-16T19:56:20.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:20.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:20.894 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:56:20.937 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:56:20.937 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:56:21.001 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:56:21.001 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:56:21.001 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:56:21.001 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:56:21.002 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:56:21.002 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:56:21.002 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:21.002 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:21.003 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:56:21.003 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:21.003 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:56:21.003 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:21.004 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:21.004 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:56:21.004 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:21.004 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:56:21.005 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:21.005 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:56:21.005 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:21.005 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:21.005 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:21.006 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:21.006 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:56:21.006 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:21.006 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:21.007 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:56:21.007 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:21.007 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:21.007 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:21.007 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:56:21.008 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:21.008 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:21.008 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:21.009 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:21.009 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:56:21.009 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:21.009 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:21.009 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.725649_200.1", 2021-09-16T19:56:21.010 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:21.010 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:21.010 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:21.010 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.725649Z", 2021-09-16T19:56:21.011 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:21.011 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:21.011 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:21.011 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:21.011 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:56:21.012 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:21.012 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:21.012 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:56:21.012 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:21.012 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:21.013 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:21.013 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:56:21.013 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:21.013 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:21.014 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:21.014 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:56:21.014 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:21.014 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:56:21.015 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:56:21.015 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:56:21.015 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:21.015 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:56:21.016 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.725649_200.1 2021-09-16T19:56:26.017 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:26.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:26.019 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:56:26.083 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:56:26.084 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:56:26.348 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:56:26.349 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:56:26.349 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:56:26.349 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:56:26.349 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:56:26.349 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:56:26.350 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:26.350 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:26.350 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:56:26.350 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:26.351 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:56:26.351 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:26.351 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:26.351 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:56:26.352 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:26.352 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:56:26.352 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:26.352 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:56:26.353 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:26.353 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:26.353 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:26.353 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:26.353 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:56:26.354 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:26.354 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:26.354 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:56:26.354 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:26.355 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:26.355 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:26.355 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:56:26.355 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:26.356 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:26.356 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:26.356 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:26.356 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:56:26.356 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:26.357 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:26.357 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:56:26.357 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:26.357 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:26.357 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:26.358 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:56:26.358 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:26.358 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:26.359 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:26.359 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:26.359 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:56:26.359 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:26.359 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:26.360 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:56:26.360 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:26.360 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:26.360 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:26.361 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:56:26.361 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:26.361 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:26.361 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:26.361 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:56:26.362 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:26.362 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:56:26.362 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:56:26.362 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:56:26.363 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:26.363 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:56:26.364 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:56:31.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:31.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:31.367 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:56:31.411 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:56:31.411 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:56:31.479 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:56:31.480 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:56:31.480 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:56:31.480 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:56:31.480 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:56:31.481 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:56:31.481 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:31.481 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:31.481 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:56:31.482 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:31.482 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:56:31.482 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:31.482 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:31.483 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:56:31.483 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:31.483 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:56:31.483 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:31.484 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:56:31.484 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:31.484 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:31.484 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:31.484 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:31.485 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:56:31.485 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:31.485 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:31.485 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:56:31.486 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:31.486 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:31.486 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:31.486 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:56:31.487 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:31.487 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:31.487 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:31.487 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:31.487 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:56:31.488 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:31.488 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:31.488 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:56:31.488 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:31.489 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:31.489 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:31.489 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:56:31.489 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:31.490 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:31.490 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:31.490 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:31.490 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:56:31.490 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:31.491 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:31.491 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:56:31.491 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:31.491 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:31.491 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:31.492 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:56:31.492 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:31.492 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:31.492 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:31.493 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:56:31.493 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:31.493 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:56:31.493 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:56:31.493 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:56:31.494 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:31.494 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:56:31.495 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:56:36.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:36.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:36.498 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:56:36.541 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:56:36.541 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:56:36.612 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:56:36.612 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:56:36.612 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:56:36.612 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:56:36.613 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:56:36.613 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:56:36.613 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:36.613 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:36.614 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:56:36.614 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:36.614 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:56:36.614 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:36.615 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:36.615 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:56:36.615 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:36.615 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:56:36.615 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:36.616 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:56:36.616 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:36.616 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:36.616 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:36.617 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:36.617 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:56:36.617 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:36.617 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:36.617 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:56:36.618 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:36.618 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:36.618 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:36.618 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:56:36.619 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:36.619 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:36.619 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:36.619 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:36.620 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:56:36.620 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:36.620 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:36.620 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.538756_188.1", 2021-09-16T19:56:36.620 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:36.621 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:36.621 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:36.621 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.538756Z", 2021-09-16T19:56:36.621 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:36.622 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:36.622 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:36.622 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:36.622 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:56:36.622 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:36.623 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:36.623 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:56:36.623 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:36.623 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:36.624 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:36.624 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:56:36.624 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:36.624 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:36.624 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:36.625 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:56:36.625 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:36.625 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:56:36.625 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:56:36.626 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:56:36.626 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:36.626 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:56:36.627 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.538756_188.1 2021-09-16T19:56:41.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:41.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:41.630 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:56:41.673 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:56:41.674 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:56:41.731 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:56:41.731 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:56:41.731 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:56:41.732 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:56:41.732 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:56:41.732 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:56:41.732 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:41.732 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:41.733 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:56:41.733 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:41.733 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:56:41.733 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:41.734 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:41.734 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:56:41.734 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:41.734 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:56:41.734 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:41.735 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:56:41.735 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:41.735 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:41.735 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:41.736 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:41.736 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:56:41.736 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:41.736 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:41.736 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:56:41.737 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:41.737 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:41.737 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:41.737 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:56:41.738 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:41.738 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:41.738 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:41.738 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:41.739 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:56:41.739 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:41.739 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:41.739 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:56:41.739 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:41.740 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:41.740 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:41.740 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:56:41.740 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:41.741 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:41.741 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:41.741 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:41.741 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:56:41.741 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:41.742 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:41.742 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:56:41.742 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:41.742 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:41.742 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:41.743 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:56:41.743 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:41.743 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:41.743 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:41.744 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:56:41.744 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:41.744 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:56:41.744 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:56:41.744 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:56:41.745 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:41.745 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:56:41.746 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.513492_186.1 2021-09-16T19:56:46.747 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:46.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:46.749 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:56:46.797 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:56:46.797 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:56:46.857 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:56:46.858 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:56:46.858 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:56:46.858 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:56:46.858 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:56:46.859 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:56:46.859 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:46.859 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:46.859 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:56:46.859 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:46.860 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:56:46.860 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:46.860 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:46.860 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:56:46.861 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:46.861 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:56:46.861 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:46.861 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:56:46.861 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:46.862 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:46.862 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:46.862 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:46.862 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:56:46.863 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:46.863 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:46.863 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:56:46.863 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:46.864 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:46.864 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:46.864 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:56:46.864 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:46.864 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:46.865 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:46.865 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:46.865 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:56:46.865 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:46.866 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:46.866 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.517759_187.1", 2021-09-16T19:56:46.866 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:46.866 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:46.866 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:46.867 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.517759Z", 2021-09-16T19:56:46.867 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:46.867 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:46.867 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:46.867 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:46.868 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:56:46.868 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:46.868 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:46.868 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:56:46.869 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:46.869 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:46.869 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:46.869 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:56:46.869 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:46.870 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:46.870 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:46.870 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:56:46.870 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:46.871 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:56:46.871 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:56:46.871 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:56:46.871 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:46.872 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:56:46.873 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.517759_187.1 2021-09-16T19:56:51.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:51.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:51.876 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:56:51.928 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:56:51.928 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:56:52.009 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:56:52.009 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:56:52.009 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:56:52.010 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:56:52.010 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:56:52.010 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:56:52.010 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:52.011 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:52.011 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:56:52.011 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:52.011 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:56:52.011 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:52.012 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:52.012 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:56:52.012 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:52.012 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:56:52.012 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:52.013 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:56:52.013 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:52.013 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:52.013 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:52.014 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:52.014 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:56:52.014 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:52.014 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:52.014 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:56:52.015 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:52.015 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:52.015 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:52.015 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:56:52.015 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:52.016 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:52.016 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:52.016 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:52.016 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:56:52.017 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:52.017 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:52.017 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.725649_200.1", 2021-09-16T19:56:52.017 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:52.017 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:52.018 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:52.018 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.725649Z", 2021-09-16T19:56:52.018 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:52.018 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:52.018 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:52.019 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:52.019 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:56:52.019 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:52.019 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:52.019 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:56:52.020 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:52.020 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:52.020 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:52.020 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:56:52.021 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:52.021 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:52.021 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:52.021 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:56:52.021 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:52.022 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:56:52.022 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:56:52.022 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:56:52.022 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:52.022 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:56:52.024 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.725649_200.1 2021-09-16T19:56:57.025 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:57.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:56:57.026 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:56:57.184 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:56:57.185 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:56:57.581 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:56:57.582 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:56:57.582 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:56:57.582 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:56:57.582 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:56:57.582 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:56:57.583 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:57.583 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:57.583 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:56:57.583 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:57.584 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:56:57.584 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:57.584 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:57.584 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:56:57.584 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:57.585 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:56:57.585 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:57.585 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:56:57.585 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:57.586 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:57.586 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:57.586 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:57.586 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:56:57.586 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:57.587 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:57.587 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:56:57.587 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:57.587 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:57.588 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:57.588 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:56:57.588 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:57.588 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:57.588 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:57.589 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:57.589 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:56:57.589 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:57.589 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:57.590 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.665525_196.1", 2021-09-16T19:56:57.590 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:57.590 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:57.590 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:57.591 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.665525Z", 2021-09-16T19:56:57.591 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:57.591 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:57.591 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:57.592 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:56:57.592 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:56:57.592 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:56:57.592 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:56:57.593 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:56:57.593 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:56:57.593 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:56:57.593 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:56:57.594 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:56:57.594 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:56:57.594 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:57.594 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:57.595 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:56:57.595 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:56:57.595 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:56:57.595 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:56:57.596 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:56:57.596 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:56:57.596 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:56:57.597 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.665525_196.1 2021-09-16T19:57:02.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:02.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:02.600 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:57:02.661 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:57:02.662 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:57:02.783 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:57:02.784 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:57:02.784 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:57:02.784 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:57:02.784 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:57:02.785 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:57:02.785 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:02.785 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:02.785 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:57:02.786 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:02.787 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:57:02.787 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:02.787 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:02.787 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:57:02.788 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:02.788 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:57:02.789 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:02.789 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:57:02.789 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:02.790 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:02.790 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:02.791 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:02.791 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:57:02.792 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:02.792 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:02.793 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:57:02.793 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:02.793 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:02.794 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:02.794 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:57:02.795 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:02.795 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:02.796 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:02.796 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:02.796 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:57:02.797 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:02.797 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:02.797 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:57:02.798 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:02.798 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:02.799 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:02.799 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:57:02.799 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:02.800 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:02.800 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:02.800 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:02.801 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:57:02.801 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:02.801 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:02.801 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:57:02.801 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:02.802 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:02.802 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:02.802 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:57:02.802 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:02.802 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:02.803 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:02.803 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:57:02.803 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:02.803 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:57:02.804 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:57:02.804 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:57:02.804 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:02.805 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:57:02.806 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.543574_190.1 2021-09-16T19:57:07.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:07.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:07.809 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:57:07.874 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:57:07.875 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:57:08.067 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:57:08.068 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:57:08.068 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:57:08.068 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:57:08.068 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:57:08.068 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:57:08.069 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:08.069 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:08.069 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:57:08.069 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:08.070 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:57:08.070 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:08.070 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:08.070 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:57:08.071 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:08.071 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:57:08.071 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:08.071 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:57:08.071 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:08.072 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:08.072 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:08.072 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:08.072 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:57:08.073 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:08.073 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:08.073 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:57:08.073 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:08.074 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:08.074 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:08.074 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:57:08.074 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:08.074 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:08.075 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:08.075 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:08.075 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:57:08.075 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:08.076 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:08.076 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:57:08.076 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:08.076 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:08.077 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:08.077 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:57:08.077 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:08.077 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:08.077 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:08.078 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:08.078 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:57:08.078 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:08.078 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:08.078 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:57:08.079 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:08.079 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:08.079 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:08.079 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:57:08.080 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:08.080 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:08.081 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:08.081 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:57:08.082 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:08.082 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:57:08.082 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:57:08.082 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:57:08.082 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:08.083 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:57:08.084 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.543574_190.1 2021-09-16T19:57:13.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:13.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:13.086 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:57:13.258 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:57:13.258 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:57:13.311 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:57:13.311 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:57:13.311 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:57:13.312 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:57:13.312 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:57:13.312 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:57:13.312 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:13.313 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:13.313 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:57:13.313 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:13.313 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:57:13.314 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:13.314 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:13.314 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:57:13.315 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:13.315 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:57:13.315 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:13.315 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:57:13.315 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:13.316 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:13.316 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:13.316 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:13.316 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:57:13.317 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:13.317 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:13.317 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:57:13.317 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:13.318 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:13.318 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:13.318 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:57:13.318 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:13.319 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:13.319 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:13.319 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:13.319 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:57:13.319 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:13.320 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:13.320 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:57:13.320 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:13.320 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:13.321 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:13.321 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:57:13.321 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:13.321 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:13.321 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:13.322 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:13.322 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:57:13.322 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:13.322 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:13.322 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:57:13.323 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:13.323 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:13.323 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:13.323 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:57:13.323 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:13.324 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:13.324 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:13.324 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:57:13.324 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:13.325 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:57:13.325 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:57:13.325 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:57:13.325 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:13.325 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:57:13.327 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.543574_190.1 2021-09-16T19:57:18.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:18.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:18.329 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:57:18.372 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:57:18.372 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:57:18.435 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:57:18.436 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:57:18.436 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:57:18.436 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:57:18.436 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:57:18.437 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:57:18.437 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:18.437 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:18.437 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:57:18.438 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:18.438 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:57:18.438 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:18.438 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:18.438 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:57:18.439 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:18.439 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:57:18.439 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:18.439 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:57:18.440 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:18.440 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:18.440 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:18.440 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:18.440 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:57:18.441 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:18.441 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:18.441 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:57:18.441 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:18.441 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:18.442 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:18.442 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:57:18.442 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:18.442 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:18.443 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:18.444 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:18.444 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:57:18.444 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:18.445 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:18.445 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:57:18.446 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:18.446 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:18.446 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:18.447 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:57:18.447 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:18.447 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:18.448 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:18.448 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:18.449 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:57:18.449 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:18.449 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:18.450 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:57:18.450 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:18.450 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:18.451 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:18.451 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:57:18.451 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:18.451 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:18.452 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:18.452 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:57:18.452 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:18.452 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:57:18.453 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:57:18.453 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:57:18.453 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:18.453 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:57:18.455 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.543574_190.1 2021-09-16T19:57:23.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:23.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:23.458 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:57:23.502 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:57:23.502 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:57:23.574 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:57:23.574 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:57:23.575 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:57:23.575 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:57:23.575 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:57:23.575 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:57:23.575 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:23.576 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:23.576 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:57:23.576 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:23.576 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:57:23.577 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:23.577 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:23.577 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:57:23.577 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:23.577 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:57:23.578 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:23.578 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:57:23.578 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:23.578 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:23.579 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:23.579 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:23.579 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:57:23.579 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:23.579 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:23.580 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:57:23.580 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:23.580 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:23.580 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:23.581 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:57:23.581 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:23.581 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:23.581 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:23.581 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:23.582 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:57:23.582 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:23.582 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:23.582 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.665525_196.1", 2021-09-16T19:57:23.583 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:23.583 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:23.583 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:23.583 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.665525Z", 2021-09-16T19:57:23.583 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:23.584 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:23.584 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:23.584 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:23.584 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:57:23.584 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:23.585 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:23.585 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:57:23.585 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:23.585 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:23.586 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:23.586 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:57:23.586 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:23.586 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:23.586 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:23.587 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:57:23.587 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:23.587 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:57:23.587 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:57:23.587 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:57:23.588 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:23.588 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:57:23.589 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.665525_196.1 2021-09-16T19:57:28.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:28.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:28.592 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:57:28.729 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:57:28.729 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:57:28.787 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:57:28.787 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:57:28.787 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:57:28.788 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:57:28.788 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:57:28.788 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:57:28.788 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:28.788 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:28.789 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:57:28.789 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:28.789 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:57:28.789 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:28.790 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:28.790 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:57:28.790 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:28.790 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:57:28.790 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:28.791 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:57:28.791 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:28.791 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:28.791 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:28.792 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:28.792 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:57:28.792 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:28.792 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:28.792 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:57:28.793 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:28.793 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:28.793 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:28.793 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:57:28.794 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:28.794 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:28.794 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:28.794 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:28.795 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:57:28.795 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:28.795 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:28.796 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:57:28.796 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:28.796 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:28.797 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:28.797 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:57:28.797 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:28.798 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:28.798 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:28.799 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:28.799 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:57:28.799 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:28.800 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:28.800 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:57:28.801 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:28.801 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:28.802 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:28.802 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:57:28.802 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:28.803 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:28.803 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:28.803 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:57:28.804 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:28.804 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:57:28.805 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:57:28.805 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:57:28.806 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:28.806 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:57:28.808 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.513492_186.1 2021-09-16T19:57:33.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:33.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:33.811 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:57:33.860 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:57:33.860 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:57:33.934 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:57:33.934 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:57:33.934 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:57:33.935 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:57:33.935 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:57:33.935 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:57:33.935 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:33.936 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:33.936 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:57:33.936 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:33.936 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:57:33.936 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:33.937 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:33.937 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:57:33.937 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:33.937 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:57:33.937 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:33.938 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:57:33.938 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:33.938 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:33.938 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:33.939 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:33.939 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:57:33.939 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:33.939 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:33.939 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:57:33.940 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:33.940 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:33.940 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:33.940 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:57:33.940 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:33.941 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:33.941 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:33.941 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:33.941 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:57:33.941 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:33.942 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:33.942 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:57:33.942 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:33.942 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:33.942 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:33.943 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:57:33.943 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:33.943 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:33.943 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:33.944 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:33.944 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:57:33.944 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:33.944 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:33.944 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:57:33.945 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:33.945 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:33.945 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:33.945 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:57:33.945 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:33.946 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:33.946 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:33.946 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:57:33.946 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:33.946 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:57:33.947 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:57:33.947 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:57:33.947 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:33.947 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:57:33.948 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.488881_181.1 2021-09-16T19:57:38.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:38.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:38.951 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:57:39.017 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:57:39.017 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:57:39.436 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:57:39.436 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:57:39.436 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:57:39.436 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:57:39.437 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:57:39.437 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:57:39.437 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:39.437 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:39.438 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:57:39.438 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:39.438 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:57:39.438 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:39.438 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:39.439 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:57:39.439 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:39.439 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:57:39.439 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:39.440 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:57:39.440 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:39.440 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:39.440 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:39.440 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:39.440 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:57:39.441 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:39.441 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:39.441 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:57:39.441 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:39.441 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:39.442 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:39.442 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:57:39.442 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:39.442 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:39.443 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:39.443 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:39.443 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:57:39.443 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:39.443 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:39.444 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.665525_196.1", 2021-09-16T19:57:39.444 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:39.444 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:39.444 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:39.445 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.665525Z", 2021-09-16T19:57:39.445 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:39.445 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:39.445 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:39.446 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:39.446 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:57:39.446 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:39.446 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:39.446 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:57:39.447 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:39.447 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:39.447 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:39.447 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:57:39.447 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:39.448 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:39.448 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:39.448 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:57:39.448 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:39.448 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:57:39.449 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:57:39.449 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:57:39.449 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:39.449 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:57:39.450 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.665525_196.1 2021-09-16T19:57:44.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:44.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:44.453 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:57:44.496 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:57:44.497 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:57:44.577 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:57:44.577 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:57:44.578 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:57:44.578 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:57:44.579 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:57:44.579 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:57:44.580 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:44.581 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:44.581 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:57:44.582 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:44.583 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:57:44.584 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:44.585 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:44.585 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:57:44.586 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:44.586 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:57:44.587 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:44.588 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:57:44.588 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:44.589 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:44.589 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:44.590 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:44.591 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:57:44.592 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:44.593 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:44.593 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:57:44.594 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:44.594 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:44.595 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:44.595 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:57:44.596 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:44.597 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:44.597 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:44.598 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:44.598 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:57:44.599 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:44.599 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:44.600 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.517759_187.1", 2021-09-16T19:57:44.600 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:44.601 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:44.602 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:44.602 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.517759Z", 2021-09-16T19:57:44.603 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:44.603 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:44.604 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:44.604 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:44.605 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:57:44.605 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:44.606 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:44.606 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:57:44.607 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:44.608 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:44.608 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:44.608 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:57:44.609 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:44.609 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:44.610 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:44.610 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:57:44.611 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:44.611 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:57:44.612 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:57:44.612 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:57:44.613 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:44.613 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:57:44.616 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.517759_187.1 2021-09-16T19:57:49.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:49.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:49.620 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:57:49.682 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:57:49.682 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:57:49.896 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:57:49.896 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:57:49.897 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:57:49.897 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:57:49.897 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:57:49.897 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:57:49.897 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:49.898 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:49.898 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:57:49.898 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:49.898 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:57:49.899 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:49.899 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:49.899 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:57:49.899 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:49.899 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:57:49.900 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:49.900 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:57:49.900 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:49.901 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:49.902 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:49.902 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:49.902 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:57:49.903 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:49.903 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:49.903 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:57:49.903 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:49.904 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:49.904 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:49.904 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:57:49.904 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:49.905 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:49.905 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:49.905 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:49.905 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:57:49.905 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:49.906 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:49.906 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:57:49.906 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:49.906 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:49.907 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:49.907 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:57:49.907 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:49.908 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:49.908 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:49.908 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:49.908 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:57:49.909 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:49.909 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:49.909 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:57:49.909 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:49.910 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:49.910 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:49.910 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:57:49.910 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:49.911 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:49.911 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:49.911 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:57:49.911 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:49.911 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:57:49.912 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:57:49.912 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:57:49.912 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:49.912 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:57:49.914 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.488881_181.1 2021-09-16T19:57:54.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:54.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:57:54.917 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:57:54.981 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:57:54.981 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:57:55.041 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:57:55.042 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:57:55.042 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:57:55.042 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:57:55.042 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:57:55.043 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:57:55.043 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:55.043 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:55.043 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:57:55.044 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:55.044 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:57:55.044 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:55.044 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:55.045 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:57:55.045 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:55.045 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:57:55.045 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:55.045 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:57:55.046 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:55.046 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:55.046 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:55.046 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:55.047 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:57:55.047 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:55.047 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:55.047 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:57:55.048 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:55.048 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:55.048 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:55.048 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:57:55.048 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:55.049 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:55.049 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:55.049 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:55.049 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:57:55.050 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:55.050 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:55.050 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:57:55.050 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:55.051 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:55.051 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:55.051 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:57:55.051 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:55.052 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:55.052 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:55.052 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:57:55.052 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:57:55.052 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:57:55.053 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:57:55.053 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:57:55.053 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:57:55.053 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:57:55.054 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:57:55.054 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:57:55.054 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:57:55.054 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:55.055 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:55.055 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:57:55.055 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:57:55.055 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:57:55.056 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:57:55.056 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:57:55.056 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:57:55.056 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:57:55.057 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.543574_190.1 2021-09-16T19:58:00.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:00.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:00.060 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:58:00.102 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:58:00.103 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:58:00.158 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:58:00.158 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:58:00.159 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:58:00.159 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:58:00.159 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:58:00.159 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:58:00.160 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:00.160 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:00.160 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:58:00.160 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:00.161 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:58:00.161 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:00.161 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:00.161 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:58:00.162 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:00.162 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:58:00.162 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:00.163 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:58:00.163 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:00.163 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:00.163 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:00.164 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:00.164 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:58:00.164 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:00.164 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:00.164 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:58:00.165 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:00.165 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:00.165 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:00.165 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:58:00.166 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:00.166 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:00.166 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:00.166 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:00.166 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:58:00.167 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:00.167 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:00.167 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:58:00.167 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:00.168 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:00.168 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:00.168 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:58:00.168 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:00.168 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:00.169 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:00.169 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:00.169 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:58:00.169 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:00.169 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:00.170 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:58:00.170 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:00.170 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:00.170 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:00.171 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:58:00.171 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:00.171 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:00.171 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:00.171 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:58:00.172 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:00.172 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:58:00.172 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:58:00.174 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:58:00.174 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:00.174 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:58:00.177 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:58:05.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:05.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:05.180 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:58:05.223 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:58:05.223 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:58:05.288 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:58:05.378 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:58:05.379 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:58:05.379 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:58:05.379 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:58:05.379 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:58:05.380 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:05.380 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:05.380 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:58:05.380 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:05.381 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:58:05.381 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:05.381 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:05.381 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:58:05.381 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:05.382 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:58:05.382 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:05.382 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:58:05.383 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:05.383 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:05.383 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:05.383 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:05.383 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:58:05.384 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:05.384 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:05.384 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:58:05.384 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:05.385 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:05.385 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:05.385 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:58:05.385 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:05.385 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:05.386 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:05.386 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:05.386 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:58:05.386 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:05.386 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:05.387 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:58:05.387 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:05.387 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:05.387 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:05.388 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:58:05.388 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:05.388 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:05.388 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:05.388 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:05.389 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:58:05.389 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:05.389 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:05.389 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:58:05.389 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:05.390 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:05.390 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:05.390 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:58:05.390 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:05.390 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:05.391 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:05.391 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:58:05.391 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:05.391 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:58:05.392 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:58:05.392 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:58:05.392 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:05.392 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:58:05.394 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:58:10.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:10.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:10.396 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:58:10.441 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:58:10.441 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:58:10.513 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:58:10.514 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:58:10.514 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:58:10.514 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:58:10.514 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:58:10.515 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:58:10.515 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:10.515 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:10.515 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:58:10.516 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:10.516 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:58:10.516 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:10.516 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:10.516 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:58:10.517 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:10.517 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:58:10.517 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:10.517 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:58:10.518 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:10.518 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:10.518 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:10.518 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:10.519 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:58:10.519 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:10.519 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:10.519 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:58:10.520 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:10.520 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:10.520 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:10.520 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:58:10.520 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:10.521 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:10.521 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:10.521 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:10.521 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:58:10.522 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:10.522 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:10.522 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.508383_184.1", 2021-09-16T19:58:10.522 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:10.523 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:10.523 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:10.523 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.508383Z", 2021-09-16T19:58:10.523 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:10.524 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:10.524 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:10.524 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:10.524 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:58:10.524 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:10.525 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:10.525 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:58:10.525 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:10.525 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:10.526 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:10.526 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:58:10.526 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:10.526 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:10.526 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:10.527 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:58:10.527 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:10.527 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:58:10.527 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:58:10.528 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:58:10.528 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:10.528 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:58:10.529 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.508383_184.1 2021-09-16T19:58:15.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:15.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:15.532 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:58:15.586 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:58:15.587 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:58:15.648 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:58:15.648 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:58:15.648 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:58:15.648 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:58:15.649 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:58:15.649 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:58:15.649 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:15.649 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:15.650 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:58:15.650 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:15.650 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:58:15.651 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:15.651 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:15.651 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:58:15.651 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:15.652 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:58:15.652 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:15.653 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:58:15.653 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:15.653 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:15.653 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:15.654 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:15.654 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:58:15.654 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:15.654 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:15.655 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:58:15.655 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:15.655 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:15.655 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:15.656 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:58:15.656 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:15.656 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:15.656 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:15.656 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:15.657 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:58:15.657 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:15.657 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:15.657 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.508383_184.1", 2021-09-16T19:58:15.658 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:15.658 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:15.658 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:15.659 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.508383Z", 2021-09-16T19:58:15.659 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:15.659 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:15.659 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:15.660 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:15.660 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:58:15.660 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:15.660 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:15.660 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:58:15.661 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:15.661 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:15.661 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:15.661 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:58:15.662 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:15.662 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:15.662 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:15.662 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:58:15.662 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:15.663 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:58:15.663 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:58:15.663 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:58:15.663 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:15.664 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:58:15.665 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.508383_184.1 2021-09-16T19:58:20.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:20.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:20.667 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:58:20.710 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:58:20.711 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:58:20.789 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:58:20.790 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:58:20.790 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:58:20.790 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:58:20.791 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:58:20.791 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:58:20.791 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:20.791 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:20.792 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:58:20.792 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:20.792 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:58:20.792 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:20.792 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:20.793 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:58:20.793 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:20.793 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:58:20.793 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:20.794 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:58:20.794 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:20.794 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:20.794 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:20.795 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:20.795 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:58:20.795 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:20.795 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:20.795 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:58:20.796 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:20.796 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:20.796 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:20.796 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:58:20.797 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:20.797 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:20.797 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:20.797 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:20.797 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:58:20.798 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:20.798 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:20.798 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.542659_189.1", 2021-09-16T19:58:20.798 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:20.798 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:20.799 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:20.799 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.542659Z", 2021-09-16T19:58:20.799 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:20.799 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:20.799 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:20.800 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:20.800 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:58:20.800 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:20.800 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:20.801 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:58:20.801 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:20.801 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:20.801 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:20.801 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:58:20.802 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:20.802 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:20.802 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:20.802 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:58:20.802 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:20.803 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:58:20.803 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:58:20.803 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:58:20.803 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:20.804 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:58:20.805 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.542659_189.1 2021-09-16T19:58:25.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:25.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:25.807 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:58:25.886 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:58:25.887 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:58:26.346 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:58:26.346 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:58:26.346 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:58:26.347 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:58:26.347 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:58:26.347 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:58:26.347 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:26.348 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:26.348 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:58:26.348 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:26.348 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:58:26.348 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:26.349 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:26.349 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:58:26.349 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:26.349 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:58:26.350 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:26.350 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:58:26.350 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:26.350 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:26.350 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:26.351 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:26.351 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:58:26.351 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:26.351 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:26.352 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:58:26.352 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:26.352 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:26.352 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:26.352 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:58:26.353 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:26.353 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:26.353 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:26.353 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:26.354 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:58:26.354 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:26.354 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:26.354 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T19:58:26.355 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:26.355 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:26.355 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:26.355 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T19:58:26.355 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:26.356 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:26.356 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:26.356 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:26.357 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:58:26.357 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:26.357 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:26.357 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:58:26.357 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:26.358 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:26.358 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:26.358 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:58:26.358 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:26.359 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:26.359 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:26.359 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:58:26.359 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:26.359 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:58:26.360 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:58:26.360 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:58:26.360 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:26.360 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:58:26.361 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.488881_181.1 2021-09-16T19:58:31.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:31.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:31.364 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:58:31.420 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:58:31.420 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:58:31.554 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:58:31.554 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:58:31.555 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:58:31.555 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:58:31.555 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:58:31.556 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:58:31.556 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:31.556 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:31.556 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:58:31.557 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:31.557 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:58:31.557 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:31.557 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:31.558 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:58:31.558 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:31.558 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:58:31.558 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:31.559 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:58:31.559 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:31.559 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:31.559 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:31.560 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:31.560 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:58:31.560 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:31.560 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:31.561 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:58:31.561 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:31.561 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:31.562 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:31.562 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:58:31.562 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:31.562 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:31.562 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:31.563 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:31.563 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:58:31.563 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:31.564 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:31.564 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:58:31.564 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:31.564 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:31.564 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:31.565 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:58:31.565 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:31.565 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:31.565 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:31.565 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:31.566 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:58:31.566 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:31.566 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:31.566 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:58:31.566 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:31.567 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:31.567 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:31.567 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:58:31.567 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:31.568 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:31.568 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:31.568 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:58:31.568 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:31.568 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:58:31.569 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:58:31.569 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:58:31.569 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:31.570 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:58:31.571 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:58:36.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:36.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:36.574 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:58:36.657 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:58:36.657 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:58:36.899 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:58:36.899 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:58:36.899 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:58:36.900 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:58:36.900 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:58:36.900 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:58:36.900 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:36.901 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:36.901 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:58:36.901 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:36.901 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:58:36.901 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:36.902 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:36.902 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:58:36.902 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:36.902 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:58:36.903 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:36.903 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:58:36.903 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:36.903 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:36.903 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:36.904 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:36.904 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:58:36.904 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:36.904 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:36.905 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:58:36.905 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:36.905 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:36.905 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:36.905 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:58:36.906 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:36.906 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:36.906 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:36.906 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:36.906 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:58:36.907 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:36.907 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:36.907 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:58:36.908 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:36.908 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:36.908 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:36.908 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:58:36.909 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:36.909 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:36.909 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:36.909 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:36.910 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:58:36.910 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:36.910 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:36.910 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:58:36.910 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:36.911 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:36.911 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:36.911 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:58:36.911 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:36.911 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:36.912 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:36.912 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:58:36.912 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:36.912 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:58:36.913 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:58:36.913 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:58:36.913 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:36.913 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:58:36.914 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.543574_190.1 2021-09-16T19:58:41.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:41.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:41.918 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:58:41.974 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:58:41.974 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:58:42.042 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:58:42.042 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:58:42.043 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:58:42.043 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:58:42.043 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:58:42.044 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:58:42.044 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:42.045 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:42.045 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:58:42.045 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:42.046 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:58:42.046 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:42.047 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:42.047 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:58:42.048 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:42.048 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:58:42.048 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:42.049 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:58:42.049 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:42.050 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:42.050 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:42.050 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:42.051 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:58:42.051 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:42.052 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:42.052 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:58:42.052 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:42.053 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:42.053 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:42.054 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:58:42.054 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:42.054 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:42.056 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:42.056 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:42.057 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:58:42.057 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:42.058 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:42.058 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.492123_183.1", 2021-09-16T19:58:42.059 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:42.059 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:42.059 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:42.060 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.492123Z", 2021-09-16T19:58:42.060 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:42.061 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:42.061 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:42.062 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:42.062 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:58:42.062 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:42.063 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:42.063 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:58:42.064 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:42.064 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:42.065 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:42.065 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:58:42.065 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:42.066 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:42.066 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:42.067 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:58:42.067 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:42.068 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:58:42.068 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:58:42.068 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:58:42.069 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:42.069 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:58:42.071 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.492123_183.1 2021-09-16T19:58:47.072 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:47.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:47.073 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:58:47.119 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:58:47.120 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:58:47.175 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:58:47.175 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:58:47.176 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:58:47.176 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:58:47.176 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:58:47.176 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:58:47.177 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:47.177 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:47.177 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:58:47.177 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:47.178 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:58:47.178 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:47.178 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:47.178 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:58:47.178 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:47.179 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:58:47.179 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:47.179 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:58:47.179 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:47.180 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:47.180 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:47.180 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:47.180 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:58:47.181 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:47.181 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:47.181 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:58:47.181 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:47.181 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:47.182 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:47.182 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:58:47.182 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:47.183 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:47.183 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:47.183 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:47.183 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:58:47.184 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:47.184 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:47.184 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.492123_183.1", 2021-09-16T19:58:47.184 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:47.184 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:47.185 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:47.185 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.492123Z", 2021-09-16T19:58:47.185 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:47.185 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:47.185 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:47.186 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:47.186 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:58:47.186 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:47.186 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:47.187 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:58:47.187 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:47.187 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:47.187 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:47.188 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:58:47.188 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:47.188 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:47.188 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:47.188 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:58:47.189 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:47.189 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:58:47.189 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:58:47.189 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:58:47.190 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:47.190 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:58:47.191 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.492123_183.1 2021-09-16T19:58:52.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:52.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:52.194 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:58:52.241 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:58:52.241 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:58:52.304 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:58:52.304 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:58:52.304 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:58:52.305 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:58:52.305 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:58:52.305 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:58:52.305 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:52.306 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:52.306 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:58:52.306 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:52.306 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:58:52.307 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:52.307 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:52.307 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:58:52.307 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:52.308 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:58:52.308 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:52.308 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:58:52.308 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:52.308 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:52.309 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:52.309 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:52.309 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:58:52.309 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:52.309 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:52.310 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:58:52.310 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:52.310 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:52.310 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:52.311 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:58:52.311 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:52.311 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:52.311 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:52.312 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:52.312 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:58:52.312 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:52.312 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:52.313 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.538756_188.1", 2021-09-16T19:58:52.313 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:52.313 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:52.314 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:52.314 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.538756Z", 2021-09-16T19:58:52.315 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:52.315 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:52.315 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:52.315 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:52.316 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:58:52.316 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:52.316 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:52.316 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:58:52.317 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:52.317 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:52.317 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:52.317 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:58:52.318 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:52.318 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:52.318 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:52.318 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:58:52.318 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:52.319 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:58:52.319 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:58:52.319 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:58:52.319 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:52.319 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:58:52.321 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.538756_188.1 2021-09-16T19:58:57.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:57.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:58:57.323 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:58:57.372 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:58:57.373 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:58:57.444 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:58:57.445 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:58:57.445 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:58:57.445 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:58:57.445 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:58:57.446 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:58:57.446 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:57.446 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:57.446 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:58:57.447 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:57.447 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:58:57.447 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:57.447 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:57.448 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:58:57.448 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:57.448 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:58:57.448 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:57.449 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:58:57.449 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:57.449 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:57.449 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:57.450 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:57.450 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:58:57.450 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:57.450 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:57.451 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:58:57.451 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:57.451 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:57.451 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:57.452 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:58:57.452 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:57.452 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:57.452 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:57.452 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:57.453 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:58:57.453 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:57.453 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:57.453 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.614164_194.1", 2021-09-16T19:58:57.454 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:57.454 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:57.454 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:57.454 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.614164Z", 2021-09-16T19:58:57.455 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:57.455 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:57.455 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:57.455 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:58:57.456 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:58:57.456 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:58:57.456 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:58:57.456 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:58:57.456 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:58:57.457 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:58:57.457 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:58:57.457 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:58:57.457 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:58:57.458 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:57.458 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:57.458 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:58:57.458 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:58:57.459 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:58:57.459 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:58:57.459 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:58:57.459 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:58:57.459 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:58:57.461 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.614164_194.1 2021-09-16T19:59:02.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:02.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:02.463 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:59:02.512 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:59:02.512 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:59:02.595 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:59:02.596 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:59:02.596 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:59:02.596 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:59:02.596 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:59:02.597 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:59:02.597 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:02.597 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:02.597 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:59:02.598 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:02.598 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:59:02.598 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:02.598 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:02.598 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:59:02.599 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:02.599 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:59:02.599 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:02.599 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:59:02.600 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:02.600 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:02.600 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:02.600 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:02.601 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:59:02.601 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:02.601 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:02.601 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:59:02.601 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:02.602 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:02.602 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:02.602 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:59:02.602 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:02.602 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:02.603 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:02.603 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:02.603 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:59:02.603 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:02.603 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:02.604 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.492123_183.1", 2021-09-16T19:59:02.604 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:02.604 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:02.604 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:02.604 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.492123Z", 2021-09-16T19:59:02.605 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:02.605 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:02.605 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:02.605 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:02.606 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:59:02.606 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:02.606 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:02.606 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:59:02.606 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:02.607 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:02.607 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:02.607 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:59:02.607 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:02.607 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:02.608 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:02.608 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:59:02.608 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:02.608 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:59:02.608 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:59:02.609 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:59:02.609 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:02.609 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:59:02.610 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.492123_183.1 2021-09-16T19:59:07.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:07.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:07.612 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:59:07.699 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:59:07.700 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:59:08.032 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:59:08.033 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:59:08.033 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:59:08.033 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:59:08.034 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:59:08.034 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:59:08.034 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:08.034 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:08.035 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:59:08.035 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:08.035 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:59:08.035 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:08.036 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:08.036 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:59:08.036 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:08.036 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:59:08.036 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:08.037 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:59:08.037 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:08.037 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:08.038 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:08.038 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:08.038 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:59:08.038 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:08.039 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:08.039 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:59:08.039 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:08.039 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:08.039 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:08.040 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:59:08.040 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:08.040 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:08.040 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:08.040 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:08.041 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:59:08.041 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:08.041 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:08.042 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T19:59:08.042 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:08.042 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:08.042 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:08.043 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T19:59:08.043 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:08.043 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:08.043 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:08.044 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:08.044 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:59:08.044 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:08.044 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:08.044 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:59:08.045 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:08.045 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:08.045 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:08.045 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:59:08.045 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:08.046 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:08.046 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:08.046 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:59:08.046 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:08.047 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:59:08.047 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:59:08.047 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:59:08.047 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:08.047 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:59:08.049 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.489753_182.1 2021-09-16T19:59:13.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:13.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:13.051 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:59:13.214 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:59:13.214 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:59:13.361 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:59:13.362 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:59:13.362 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:59:13.362 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:59:13.363 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:59:13.363 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:59:13.363 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:13.363 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:13.363 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:59:13.364 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:13.364 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:59:13.364 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:13.364 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:13.365 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:59:13.365 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:13.365 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:59:13.365 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:13.365 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:59:13.366 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:13.366 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:13.366 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:13.366 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:13.367 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:59:13.367 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:13.367 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:13.367 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:59:13.367 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:13.368 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:13.368 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:13.368 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:59:13.368 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:13.369 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:13.369 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:13.369 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:13.369 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:59:13.369 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:13.370 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:13.370 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.765733_208.1", 2021-09-16T19:59:13.370 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:13.370 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:13.371 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:13.371 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.765733Z", 2021-09-16T19:59:13.371 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:13.371 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:13.371 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:13.372 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:13.372 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:59:13.372 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:13.372 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:13.372 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:59:13.373 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:13.373 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:13.373 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:13.373 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:59:13.374 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:13.374 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:13.374 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:13.374 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:59:13.374 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:13.375 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:59:13.375 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:59:13.375 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:59:13.375 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:13.376 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:59:13.377 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.765733_208.1 2021-09-16T19:59:18.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:18.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:18.380 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:59:18.432 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:59:18.432 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:59:18.487 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:59:18.488 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:59:18.488 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:59:18.488 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:59:18.489 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:59:18.489 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:59:18.489 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:18.489 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:18.490 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:59:18.490 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:18.490 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:59:18.490 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:18.490 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:18.491 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:59:18.491 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:18.491 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:59:18.491 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:18.492 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:59:18.492 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:18.492 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:18.492 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:18.492 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:18.493 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:59:18.493 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:18.493 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:18.493 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:59:18.494 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:18.494 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:18.494 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:18.494 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:59:18.494 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:18.495 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:18.495 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:18.495 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:18.495 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:59:18.496 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:18.496 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:18.496 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T19:59:18.496 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:18.497 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:18.497 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:18.497 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T19:59:18.497 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:18.497 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:18.498 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:18.498 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:18.498 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:59:18.498 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:18.499 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:18.499 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:59:18.499 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:18.499 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:18.499 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:18.500 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:59:18.500 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:18.500 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:18.500 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:18.501 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:59:18.501 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:18.501 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:59:18.501 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:59:18.501 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:59:18.502 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:18.502 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:59:18.503 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.543574_190.1 2021-09-16T19:59:23.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:23.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:23.506 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:59:23.551 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:59:23.552 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:59:23.605 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:59:23.606 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:59:23.606 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:59:23.606 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:59:23.607 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:59:23.607 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:59:23.607 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:23.607 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:23.608 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:59:23.608 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:23.608 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:59:23.608 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:23.609 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:23.609 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:59:23.609 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:23.609 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:59:23.609 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:23.610 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:59:23.610 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:23.610 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:23.610 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:23.611 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:23.611 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:59:23.611 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:23.611 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:23.612 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:59:23.612 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:23.612 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:23.612 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:23.612 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:59:23.613 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:23.613 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:23.613 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:23.613 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:23.614 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:59:23.614 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:23.614 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:23.614 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.517759_187.1", 2021-09-16T19:59:23.614 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:23.615 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:23.615 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:23.615 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.517759Z", 2021-09-16T19:59:23.615 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:23.616 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:23.616 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:23.616 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:23.616 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:59:23.616 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:23.617 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:23.617 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:59:23.617 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:23.617 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:23.617 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:23.618 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:59:23.618 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:23.618 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:23.618 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:23.619 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:59:23.619 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:23.619 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:59:23.619 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:59:23.619 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:59:23.620 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:23.620 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:59:23.621 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.517759_187.1 2021-09-16T19:59:28.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:28.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:28.624 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:59:28.667 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:59:28.667 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:59:28.735 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:59:28.736 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:59:28.736 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:59:28.736 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:59:28.736 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:59:28.736 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:59:28.737 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:28.737 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:28.737 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:59:28.737 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:28.738 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:59:28.738 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:28.738 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:28.738 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:59:28.738 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:28.739 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:59:28.739 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:28.739 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:59:28.739 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:28.740 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:28.740 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:28.740 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:28.740 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:59:28.740 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:28.741 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:28.741 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:59:28.741 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:28.741 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:28.741 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:28.742 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:59:28.742 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:28.742 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:28.742 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:28.743 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:28.743 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:59:28.743 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:28.743 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:28.743 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:59:28.744 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:28.744 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:28.744 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:28.744 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:59:28.745 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:28.745 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:28.745 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:28.745 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:28.745 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:59:28.746 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:28.746 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:28.746 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:59:28.746 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:28.747 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:28.747 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:28.747 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:59:28.747 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:28.747 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:28.748 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:28.748 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:59:28.748 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:28.748 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:59:28.748 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:59:28.749 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:59:28.749 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:28.749 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:59:28.750 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.513492_186.1 2021-09-16T19:59:33.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:33.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:33.753 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:59:33.802 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:59:33.802 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:59:33.870 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:59:33.871 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:59:33.871 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:59:33.871 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:59:33.871 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:59:33.872 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:59:33.872 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:33.872 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:33.872 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:59:33.872 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:33.873 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:59:33.873 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:33.873 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:33.873 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:59:33.873 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:33.874 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:59:33.874 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:33.874 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:59:33.874 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:33.875 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:33.875 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:33.875 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:33.875 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:59:33.876 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:33.876 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:33.876 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:59:33.876 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:33.876 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:33.877 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:33.877 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:59:33.877 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:33.877 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:33.877 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:33.878 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:33.878 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:59:33.878 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:33.878 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:33.878 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.727919_201.1", 2021-09-16T19:59:33.879 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:33.879 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:33.879 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:33.879 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.727919Z", 2021-09-16T19:59:33.879 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:33.880 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:33.880 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:33.880 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:33.880 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:59:33.880 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:33.881 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:33.881 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:59:33.881 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:33.881 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:33.882 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:33.882 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:59:33.882 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:33.882 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:33.882 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:33.883 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:59:33.883 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:33.883 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:59:33.883 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:59:33.884 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:59:33.884 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:33.884 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:59:33.885 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.727919_201.1 2021-09-16T19:59:38.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:38.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:38.888 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:59:39.020 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:59:39.020 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:59:39.077 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:59:39.077 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:59:39.077 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:59:39.078 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:59:39.078 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:59:39.078 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:59:39.079 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:39.079 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:39.079 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:59:39.079 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:39.079 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:59:39.080 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:39.080 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:39.080 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:59:39.080 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:39.081 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:59:39.081 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:39.081 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:59:39.081 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:39.081 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:39.082 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:39.082 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:39.082 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:59:39.082 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:39.083 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:39.083 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:59:39.083 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:39.083 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:39.084 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:39.084 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:59:39.084 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:39.084 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:39.084 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:39.085 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:39.085 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:59:39.085 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:39.085 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:39.085 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821519.035503_222.1", 2021-09-16T19:59:39.086 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:39.086 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:39.086 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:39.087 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:19.035503Z", 2021-09-16T19:59:39.087 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:39.087 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:39.087 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:39.088 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:39.088 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:59:39.088 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:39.088 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:39.089 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:59:39.089 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:39.089 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:39.089 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:39.089 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:59:39.090 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:39.090 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:39.090 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:39.090 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:59:39.091 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:39.091 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:59:39.091 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:59:39.091 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:59:39.091 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:39.092 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:59:39.093 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821519.035503_222.1 2021-09-16T19:59:44.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:44.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:44.096 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:59:44.146 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:59:44.147 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:59:44.200 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:59:44.201 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:59:44.201 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:59:44.201 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:59:44.201 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:59:44.202 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:59:44.202 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:44.202 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:44.202 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:59:44.202 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:44.203 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:59:44.203 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:44.203 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:44.203 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:59:44.204 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:44.205 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:59:44.205 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:44.206 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:59:44.206 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:44.206 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:44.206 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:44.206 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:44.207 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:59:44.207 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:44.207 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:44.207 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:59:44.208 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:44.208 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:44.208 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:44.208 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:59:44.209 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:44.209 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:44.209 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:44.209 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:44.209 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:59:44.210 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:44.210 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:44.210 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T19:59:44.210 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:44.211 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:44.211 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:44.211 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T19:59:44.211 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:44.211 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:44.212 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:44.212 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:44.212 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:59:44.212 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:44.213 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:44.213 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:59:44.213 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:44.214 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:44.214 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:44.214 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:59:44.215 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:44.215 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:44.215 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:44.216 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:59:44.216 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:44.216 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:59:44.216 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:59:44.216 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:59:44.217 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:44.217 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:59:44.219 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631821833.719264_267.1 target=1_1631821518.513492_186.1 2021-09-16T19:59:49.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... FAIL 2021-09-16T19:59:49.222 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-42 2021-09-16T19:59:49.279 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=htrfwo-43 2021-09-16T19:59:49.361 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=htrfwo-44 2021-09-16T19:59:49.802 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2021-09-16T19:59:49.802+0000 7f0e059a2700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:59:49.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:49.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:49.829 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:59:49.865 INFO:tasks.rgw.c1.client.1.smithi109.stdout:2021-09-16T19:59:49.864+0000 7f251feae700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:59:49.866 INFO:tasks.rgw.c2.client.0.smithi118.stdout:2021-09-16T19:59:49.864+0000 7f0e079a6700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:59:49.868 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T19:59:49.867+0000 7f2fd4b6b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:59:49.873 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:59:49.874 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:59:49.997 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T19:59:50.034 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T19:59:50.034 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:50.034 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:59:50.035 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T19:59:50.035 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:50.035 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:50.035 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:59:50.035 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T19:59:50.036 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:50.036 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:50.036 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631822389.517479_289.1", 2021-09-16T19:59:50.036 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:59:49.517479Z" 2021-09-16T19:59:50.037 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:50.037 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:50.037 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:59:50.037 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:17.633408Z" 2021-09-16T19:59:50.038 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:50.038 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T19:59:50.039 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T19:59:50.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:50.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:50.041 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:59:50.089 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:59:50.090 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:59:50.145 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:59:50.145 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:59:50.145 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:59:50.146 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:59:50.146 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:59:50.146 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:59:50.146 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:59:50.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:59:50.147 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:59:50.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:59:50.147 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:59:50.147 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:59:50.148 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:59:50.148 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:59:50.148 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:59:50.148 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:59:50.148 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:59:50.149 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:59:50.149 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:59:50.149 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:59:50.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:59:50.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:59:50.150 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:59:50.150 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:59:50.150 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:59:50.150 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:59:50.151 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:59:50.151 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:59:50.151 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:59:50.152 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:59:50.152 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:59:50.152 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:59:50.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:59:50.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:59:50.153 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:59:50.153 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:59:50.153 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:59:50.153 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821833.719264_267.1", 2021-09-16T19:59:50.153 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:59:50.154 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:59:50.154 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:59:50.154 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:50:33.719264Z", 2021-09-16T19:59:50.154 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:59:50.155 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:59:50.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:59:50.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:59:50.155 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:59:50.155 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:59:50.156 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:59:50.156 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:59:50.156 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:59:50.156 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:59:50.156 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:59:50.157 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:59:50.157 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:59:50.157 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:59:50.157 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:59:50.157 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:59:50.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:59:50.158 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:59:50.158 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:59:50.158 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:59:50.159 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:59:50.159 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:59:50.160 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821833.719264_267.1 2021-09-16T19:59:50.762 INFO:tasks.rgw.c1.client.1.smithi109.stdout:2021-09-16T19:59:50.761+0000 7f251769d700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:59:50.762 INFO:tasks.rgw.c1.client.1.smithi109.stdout:2021-09-16T19:59:50.762+0000 7f2512e94700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T19:59:55.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:55.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:55.163 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:59:55.211 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:59:55.212 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:59:55.267 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T19:59:55.267 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T19:59:55.267 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T19:59:55.267 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T19:59:55.268 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T19:59:55.268 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:59:55.268 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:59:55.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:59:55.269 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T19:59:55.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:59:55.269 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T19:59:55.269 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:59:55.269 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:59:55.270 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:59:55.270 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:59:55.270 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T19:59:55.270 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:59:55.271 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:59:55.271 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:59:55.271 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:59:55.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:59:55.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:59:55.272 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T19:59:55.272 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:59:55.272 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:59:55.272 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T19:59:55.273 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:59:55.273 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:59:55.273 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:59:55.273 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T19:59:55.273 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:59:55.274 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:59:55.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:59:55.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:59:55.274 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T19:59:55.275 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:59:55.275 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:59:55.275 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631822389.517479_289.1", 2021-09-16T19:59:55.275 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:59:55.275 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:59:55.276 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:59:55.276 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:59:49.517479Z", 2021-09-16T19:59:55.276 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:59:55.276 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:59:55.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:59:55.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T19:59:55.277 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T19:59:55.277 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T19:59:55.277 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T19:59:55.278 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:59:55.278 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T19:59:55.278 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T19:59:55.278 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T19:59:55.278 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:59:55.279 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T19:59:55.279 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:59:55.279 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:59:55.279 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T19:59:55.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T19:59:55.280 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T19:59:55.280 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T19:59:55.280 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T19:59:55.280 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T19:59:55.281 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T19:59:55.282 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T19:59:55.282 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T19:59:55.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:55.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T19:59:55.284 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T19:59:55.327 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T19:59:55.327 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T19:59:55.387 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T19:59:55.387 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T19:59:55.387 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T19:59:55.388 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T19:59:55.388 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T19:59:55.388 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T19:59:55.388 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:55.389 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:55.389 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T19:59:55.389 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:55.389 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T19:59:55.389 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:55.390 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:55.390 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T19:59:55.390 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:55.390 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T19:59:55.391 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:55.391 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T19:59:55.391 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:55.391 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:55.391 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:55.392 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:55.392 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T19:59:55.392 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:55.393 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:55.393 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T19:59:55.393 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:55.394 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:55.394 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:55.394 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T19:59:55.394 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:55.395 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:55.395 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:55.395 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:55.395 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T19:59:55.396 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:55.396 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:55.396 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.492123_183.1", 2021-09-16T19:59:55.396 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:55.397 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:55.397 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:55.397 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.492123Z", 2021-09-16T19:59:55.397 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:55.398 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:55.398 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:55.398 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T19:59:55.398 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T19:59:55.398 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T19:59:55.399 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T19:59:55.399 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T19:59:55.399 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T19:59:55.399 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T19:59:55.399 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T19:59:55.400 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T19:59:55.400 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T19:59:55.400 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:55.400 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:55.401 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T19:59:55.401 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T19:59:55.401 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T19:59:55.401 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T19:59:55.401 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T19:59:55.402 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T19:59:55.402 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T19:59:55.403 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.492123_183.1 2021-09-16T20:00:00.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:00.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:00.406 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:00:00.469 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:00:00.469 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:00:00.528 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:00:00.528 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:00:00.529 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:00:00.529 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:00:00.529 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:00:00.529 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:00:00.529 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:00.530 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:00.530 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:00:00.530 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:00.530 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:00:00.531 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:00.531 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:00.531 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:00:00.531 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:00.532 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:00:00.532 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:00.532 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:00:00.532 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:00.532 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:00.533 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:00.533 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:00.533 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:00:00.533 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:00.534 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:00.534 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:00:00.534 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:00.534 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:00.535 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:00.535 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:00:00.535 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:00.535 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:00.535 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:00.536 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:00.536 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:00:00.536 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:00.536 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:00.537 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T20:00:00.537 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:00.537 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:00.537 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:00.538 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T20:00:00.538 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:00.538 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:00.538 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:00.539 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:00.539 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:00:00.539 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:00.539 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:00.539 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:00:00.540 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:00.540 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:00.540 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:00.540 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:00:00.541 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:00.541 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:00.541 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:00.541 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:00:00.541 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:00.542 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:00:00.542 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:00:00.543 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:00:00.543 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:00.543 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:00:00.545 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.513492_186.1 2021-09-16T20:00:05.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:05.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:05.547 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:00:05.593 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:00:05.593 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:00:05.648 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:00:05.648 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:00:05.648 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:00:05.649 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:00:05.649 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:00:05.649 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:00:05.649 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:05.650 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:05.650 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:00:05.650 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:05.650 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:00:05.651 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:05.651 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:05.651 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:00:05.651 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:05.652 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:00:05.652 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:05.652 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:00:05.652 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:05.653 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:05.653 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:05.653 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:05.653 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:00:05.654 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:05.654 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:05.654 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:00:05.654 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:05.654 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:05.655 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:05.655 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:00:05.655 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:05.655 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:05.656 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:05.656 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:05.656 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:00:05.656 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:05.657 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:05.657 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.549156_191.1", 2021-09-16T20:00:05.657 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:05.657 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:05.658 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:05.658 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.549156Z", 2021-09-16T20:00:05.658 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:05.658 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:05.659 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:05.659 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:05.659 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:00:05.659 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:05.659 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:05.660 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:00:05.660 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:05.660 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:05.660 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:05.661 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:00:05.661 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:05.661 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:05.661 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:05.661 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:00:05.662 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:05.662 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:00:05.662 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:00:05.662 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:00:05.663 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:05.663 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:00:05.664 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.549156_191.1 2021-09-16T20:00:10.666 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:10.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:10.667 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:00:10.715 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:00:10.716 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:00:10.777 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:00:10.778 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:00:10.778 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:00:10.778 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:00:10.779 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:00:10.779 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:00:10.779 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:10.779 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:10.780 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:00:10.780 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:10.780 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:00:10.780 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:10.781 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:10.781 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:00:10.781 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:10.781 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:00:10.781 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:10.782 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:00:10.782 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:10.782 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:10.782 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:10.783 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:10.783 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:00:10.783 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:10.783 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:10.783 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:00:10.784 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:10.784 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:10.784 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:10.784 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:00:10.785 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:10.785 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:10.785 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:10.785 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:10.785 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:00:10.786 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:10.786 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:10.786 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.517759_187.1", 2021-09-16T20:00:10.786 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:10.787 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:10.787 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:10.787 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.517759Z", 2021-09-16T20:00:10.787 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:10.787 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:10.788 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:10.788 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:10.788 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:00:10.788 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:10.789 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:10.789 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:00:10.789 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:10.789 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:10.789 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:10.790 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:00:10.790 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:10.790 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:10.790 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:10.791 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:00:10.791 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:10.791 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:00:10.791 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:00:10.791 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:00:10.792 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:10.792 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:00:10.793 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.517759_187.1 2021-09-16T20:00:15.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:15.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:15.796 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:00:15.839 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:00:15.840 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:00:15.907 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:00:15.908 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:00:15.908 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:00:15.908 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:00:15.908 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:00:15.909 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:00:15.909 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:15.909 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:15.909 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:00:15.909 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:15.910 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:00:15.910 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:15.910 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:15.910 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:00:15.911 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:15.911 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:00:15.911 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:15.911 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:00:15.911 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:15.912 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:15.912 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:15.912 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:15.912 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:00:15.913 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:15.913 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:15.913 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:00:15.913 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:15.913 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:15.914 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:15.914 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:00:15.914 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:15.914 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:15.915 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:15.915 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:15.915 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:00:15.915 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:15.916 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:15.916 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.549156_191.1", 2021-09-16T20:00:15.916 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:15.916 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:15.917 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:15.917 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.549156Z", 2021-09-16T20:00:15.917 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:15.917 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:15.918 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:15.918 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:15.918 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:00:15.918 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:15.919 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:15.919 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:00:15.919 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:15.919 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:15.919 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:15.920 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:00:15.920 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:15.920 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:15.920 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:15.921 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:00:15.921 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:15.921 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:00:15.921 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:00:15.922 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:00:15.922 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:15.922 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:00:15.923 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.549156_191.1 2021-09-16T20:00:20.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:20.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:20.926 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:00:20.968 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:00:20.968 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:00:21.034 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:00:21.034 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:00:21.034 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:00:21.035 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:00:21.035 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:00:21.035 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:00:21.035 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:21.036 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:21.036 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:00:21.036 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:21.036 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:00:21.037 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:21.037 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:21.037 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:00:21.037 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:21.037 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:00:21.038 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:21.038 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:00:21.038 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:21.038 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:21.039 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:21.039 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:21.039 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:00:21.039 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:21.040 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:21.040 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:00:21.040 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:21.040 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:21.040 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:21.041 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:00:21.041 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:21.041 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:21.041 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:21.041 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:21.042 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:00:21.042 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:21.042 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:21.042 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T20:00:21.043 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:21.043 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:21.043 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:21.043 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T20:00:21.044 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:21.044 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:21.044 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:21.044 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:21.044 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:00:21.045 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:21.045 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:21.045 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:00:21.045 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:21.046 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:21.046 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:21.046 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:00:21.046 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:21.046 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:21.047 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:21.047 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:00:21.047 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:21.047 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:00:21.048 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:00:21.048 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:00:21.048 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:21.048 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:00:21.049 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.513492_186.1 2021-09-16T20:00:26.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:26.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:26.052 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:00:26.095 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:00:26.095 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:00:26.155 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:00:26.155 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:00:26.155 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:00:26.156 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:00:26.156 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:00:26.156 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:00:26.156 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:26.157 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:26.157 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:00:26.157 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:26.157 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:00:26.157 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:26.158 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:26.158 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:00:26.158 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:26.158 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:00:26.159 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:26.159 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:00:26.159 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:26.159 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:26.160 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:26.160 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:26.160 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:00:26.160 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:26.160 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:26.161 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:00:26.161 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:26.161 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:26.161 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:26.162 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:00:26.162 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:26.162 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:26.162 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:26.162 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:26.163 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:00:26.163 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:26.163 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:26.163 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T20:00:26.164 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:26.164 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:26.164 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:26.164 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T20:00:26.164 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:26.165 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:26.165 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:26.165 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:26.165 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:00:26.166 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:26.166 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:26.166 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:00:26.166 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:26.166 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:26.167 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:26.167 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:00:26.167 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:26.167 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:26.168 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:26.168 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:00:26.168 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:26.168 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:00:26.168 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:00:26.169 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:00:26.169 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:26.169 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:00:26.170 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.543574_190.1 2021-09-16T20:00:31.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:31.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:31.173 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:00:31.215 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:00:31.216 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:00:31.266 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:00:31.266 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:00:31.266 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:00:31.266 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:00:31.267 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:00:31.267 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:00:31.267 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:31.268 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:31.268 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:00:31.268 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:31.268 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:00:31.269 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:31.269 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:31.269 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:00:31.269 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:31.270 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:00:31.270 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:31.270 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:00:31.270 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:31.270 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:31.271 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:31.271 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:31.271 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:00:31.271 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:31.272 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:31.272 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:00:31.272 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:31.272 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:31.272 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:31.273 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:00:31.273 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:31.273 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:31.273 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:31.274 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:31.274 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:00:31.274 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:31.274 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:31.274 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T20:00:31.275 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:31.275 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:31.275 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:31.275 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T20:00:31.276 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:31.276 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:31.276 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:31.276 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:31.276 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:00:31.277 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:31.277 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:31.277 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:00:31.277 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:31.277 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:31.278 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:31.278 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:00:31.278 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:31.278 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:31.279 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:31.279 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:00:31.279 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:31.279 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:00:31.279 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:00:31.280 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:00:31.280 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:31.280 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:00:31.281 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.489753_182.1 2021-09-16T20:00:36.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:36.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:36.284 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:00:36.327 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:00:36.328 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:00:36.384 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:00:36.385 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:00:36.385 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:00:36.385 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:00:36.385 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:00:36.386 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:00:36.386 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:36.386 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:36.386 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:00:36.387 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:36.387 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:00:36.387 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:36.387 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:36.387 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:00:36.388 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:36.388 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:00:36.388 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:36.388 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:00:36.389 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:36.389 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:36.389 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:36.389 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:36.389 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:00:36.390 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:36.390 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:36.390 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:00:36.390 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:36.391 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:36.391 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:36.391 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:00:36.391 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:36.391 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:36.392 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:36.392 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:36.392 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:00:36.392 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:36.393 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:36.393 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.665525_196.1", 2021-09-16T20:00:36.393 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:36.393 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:36.393 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:36.394 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.665525Z", 2021-09-16T20:00:36.394 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:36.394 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:36.394 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:36.395 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:36.395 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:00:36.395 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:36.395 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:36.395 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:00:36.396 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:36.396 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:36.396 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:36.396 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:00:36.397 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:36.397 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:36.397 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:36.397 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:00:36.397 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:36.398 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:00:36.398 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:00:36.398 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:00:36.398 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:36.399 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:00:36.400 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.665525_196.1 2021-09-16T20:00:41.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:41.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:41.403 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:00:41.477 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:00:41.477 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:00:41.553 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:00:41.553 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:00:41.554 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:00:41.554 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:00:41.554 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:00:41.554 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:00:41.555 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:41.555 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:41.555 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:00:41.555 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:41.556 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:00:41.556 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:41.556 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:41.556 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:00:41.556 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:41.557 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:00:41.557 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:41.557 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:00:41.557 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:41.558 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:41.558 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:41.558 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:41.558 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:00:41.559 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:41.559 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:41.559 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:00:41.559 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:41.559 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:41.560 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:41.560 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:00:41.560 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:41.560 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:41.560 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:41.561 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:41.561 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:00:41.561 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:41.561 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:41.561 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T20:00:41.562 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:41.562 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:41.562 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:41.562 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T20:00:41.563 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:41.563 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:41.563 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:41.563 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:41.563 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:00:41.564 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:41.564 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:41.564 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:00:41.564 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:41.565 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:41.565 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:41.565 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:00:41.565 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:41.565 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:41.566 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:41.566 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:00:41.566 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:41.566 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:00:41.567 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:00:41.567 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:00:41.567 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:41.567 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:00:41.568 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.543574_190.1 2021-09-16T20:00:46.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:46.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:46.571 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:00:47.461 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:00:47.462 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:00:47.590 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:00:47.590 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:00:47.591 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:00:47.591 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:00:47.591 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:00:47.591 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:00:47.592 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:47.592 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:47.592 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:00:47.592 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:47.592 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:00:47.593 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:47.593 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:47.593 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:00:47.593 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:47.594 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:00:47.594 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:47.594 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:00:47.594 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:47.595 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:47.595 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:47.595 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:47.595 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:00:47.595 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:47.596 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:47.596 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:00:47.596 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:47.596 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:47.597 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:47.597 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:00:47.597 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:47.597 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:47.597 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:47.598 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:47.598 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:00:47.598 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:47.598 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:47.599 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.616790_195.1", 2021-09-16T20:00:47.599 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:47.599 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:47.599 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:47.599 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.616790Z", 2021-09-16T20:00:47.600 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:47.600 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:47.600 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:47.600 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:47.601 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:00:47.601 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:47.601 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:47.601 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:00:47.601 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:47.602 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:47.602 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:47.602 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:00:47.602 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:47.603 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:47.603 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:47.603 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:00:47.603 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:47.603 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:00:47.604 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:00:47.604 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:00:47.604 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:47.605 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:00:47.606 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.616790_195.1 2021-09-16T20:00:52.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:52.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:52.608 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:00:52.653 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:00:52.654 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:00:52.730 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:00:52.731 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:00:52.731 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:00:52.731 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:00:52.732 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:00:52.732 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:00:52.732 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:52.732 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:52.733 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:00:52.733 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:52.733 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:00:52.733 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:52.733 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:52.734 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:00:52.734 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:52.734 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:00:52.734 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:52.735 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:00:52.735 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:52.735 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:52.735 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:52.736 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:52.736 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:00:52.736 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:52.736 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:52.736 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:00:52.737 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:52.737 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:52.737 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:52.737 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:00:52.738 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:52.738 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:52.738 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:52.738 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:52.738 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:00:52.739 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:52.739 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:52.739 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.538756_188.1", 2021-09-16T20:00:52.739 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:52.740 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:52.740 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:52.740 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.538756Z", 2021-09-16T20:00:52.740 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:52.740 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:52.741 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:52.741 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:52.741 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:00:52.741 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:52.741 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:52.742 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:00:52.742 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:52.742 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:52.742 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:52.743 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:00:52.743 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:52.743 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:52.743 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:52.744 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:00:52.744 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:52.744 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:00:52.744 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:00:52.745 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:00:52.745 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:52.745 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:00:52.746 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.538756_188.1 2021-09-16T20:00:57.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:57.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:00:57.749 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:00:57.792 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:00:57.792 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:00:57.853 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:00:57.854 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:00:57.854 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:00:57.854 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:00:57.854 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:00:57.855 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:00:57.855 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:57.855 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:57.855 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:00:57.856 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:57.856 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:00:57.856 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:57.856 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:57.857 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:00:57.857 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:57.857 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:00:57.857 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:57.858 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:00:57.858 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:57.858 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:57.858 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:57.858 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:57.859 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:00:57.859 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:57.859 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:57.859 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:00:57.860 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:57.860 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:57.860 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:57.860 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:00:57.860 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:57.861 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:57.861 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:57.861 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:57.861 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:00:57.861 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:57.862 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:57.862 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T20:00:57.862 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:57.862 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:57.863 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:57.863 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T20:00:57.863 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:57.863 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:57.864 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:57.864 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:00:57.864 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:00:57.864 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:00:57.864 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:00:57.865 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:00:57.865 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:00:57.865 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:00:57.865 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:00:57.866 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:00:57.866 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:00:57.866 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:57.866 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:57.867 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:00:57.867 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:00:57.867 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:00:57.867 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:00:57.868 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:00:57.868 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:00:57.868 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:00:57.869 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.543574_190.1 2021-09-16T20:01:02.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:02.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:02.872 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:01:02.917 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:01:02.917 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:01:02.977 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:01:02.977 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:01:02.977 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:01:02.978 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:01:02.978 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:01:02.978 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:01:02.978 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:02.979 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:02.979 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:01:02.979 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:02.979 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:01:02.980 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:02.980 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:02.980 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:01:02.980 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:02.980 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:01:02.981 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:02.981 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:01:02.981 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:02.981 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:02.982 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:02.982 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:02.982 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:01:02.982 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:02.982 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:02.983 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:01:02.983 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:02.983 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:02.983 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:02.984 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:01:02.984 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:02.984 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:02.984 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:02.984 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:02.985 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:01:02.985 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:02.985 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:02.985 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.492123_183.1", 2021-09-16T20:01:02.985 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:02.986 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:02.986 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:02.986 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.492123Z", 2021-09-16T20:01:02.986 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:02.987 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:02.987 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:02.987 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:02.987 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:01:02.987 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:02.988 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:02.988 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:01:02.988 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:02.988 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:02.989 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:02.989 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:01:02.989 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:02.990 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:02.990 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:02.990 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:01:02.990 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:02.991 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:01:02.991 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:01:02.991 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:01:02.991 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:02.991 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:01:02.993 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.492123_183.1 2021-09-16T20:01:07.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:07.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:07.995 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:01:08.038 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:01:08.038 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:01:08.105 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:01:08.106 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:01:08.106 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:01:08.106 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:01:08.106 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:01:08.107 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:01:08.107 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:08.107 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:08.107 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:01:08.108 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:08.108 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:01:08.108 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:08.108 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:08.108 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:01:08.109 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:08.109 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:01:08.109 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:08.109 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:01:08.109 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:08.110 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:08.110 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:08.110 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:08.110 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:01:08.111 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:08.111 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:08.111 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:01:08.111 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:08.111 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:08.112 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:08.112 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:01:08.112 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:08.112 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:08.113 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:08.113 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:08.113 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:01:08.113 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:08.113 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:08.114 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T20:01:08.114 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:08.114 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:08.114 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:08.114 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T20:01:08.115 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:08.115 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:08.115 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:08.115 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:08.116 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:01:08.116 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:08.116 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:08.116 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:01:08.116 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:08.117 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:08.117 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:08.117 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:01:08.117 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:08.117 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:08.118 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:08.118 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:01:08.118 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:08.118 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:01:08.119 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:01:08.119 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:01:08.119 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:08.119 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:01:08.120 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.513492_186.1 2021-09-16T20:01:13.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:13.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:13.122 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:01:13.165 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:01:13.166 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:01:13.231 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:01:13.231 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:01:13.231 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:01:13.232 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:01:13.232 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:01:13.232 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:01:13.232 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:13.233 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:13.233 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:01:13.233 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:13.233 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:01:13.233 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:13.234 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:13.234 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:01:13.234 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:13.234 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:01:13.235 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:13.235 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:01:13.235 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:13.235 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:13.235 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:13.236 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:13.236 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:01:13.236 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:13.236 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:13.237 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:01:13.237 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:13.237 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:13.237 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:13.237 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:01:13.238 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:13.238 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:13.238 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:13.238 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:13.239 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:01:13.239 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:13.239 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:13.239 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.508383_184.1", 2021-09-16T20:01:13.239 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:13.240 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:13.240 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:13.240 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.508383Z", 2021-09-16T20:01:13.240 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:13.240 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:13.241 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:13.241 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:13.241 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:01:13.241 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:13.242 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:13.242 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:01:13.242 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:13.242 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:13.242 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:13.243 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:01:13.243 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:13.243 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:13.243 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:13.244 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:01:13.244 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:13.244 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:01:13.244 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:01:13.244 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:01:13.245 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:13.245 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:01:13.246 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.508383_184.1 2021-09-16T20:01:18.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:18.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:18.248 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:01:18.382 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:01:18.382 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:01:18.455 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:01:18.455 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:01:18.456 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:01:18.456 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:01:18.457 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:01:18.457 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:01:18.457 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:18.458 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:18.458 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:01:18.459 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:18.459 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:01:18.459 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:18.460 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:18.460 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:01:18.461 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:18.461 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:01:18.461 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:18.462 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:01:18.462 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:18.463 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:18.463 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:18.464 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:18.464 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:01:18.464 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:18.465 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:18.465 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:01:18.466 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:18.466 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:18.467 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:18.467 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:01:18.467 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:18.467 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:18.468 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:18.468 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:18.468 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:01:18.468 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:18.469 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:18.469 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.508383_184.1", 2021-09-16T20:01:18.469 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:18.469 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:18.469 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:18.470 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.508383Z", 2021-09-16T20:01:18.470 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:18.470 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:18.470 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:18.471 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:18.471 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:01:18.471 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:18.471 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:18.472 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:01:18.472 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:18.472 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:18.472 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:18.472 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:01:18.473 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:18.473 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:18.473 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:18.473 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:01:18.473 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:18.474 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:01:18.474 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:01:18.474 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:01:18.474 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:18.475 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:01:18.476 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.508383_184.1 2021-09-16T20:01:23.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:23.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:23.480 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:01:23.661 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:01:23.661 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:01:23.722 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:01:23.722 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:01:23.722 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:01:23.723 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:01:23.723 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:01:23.723 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:01:23.723 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:23.724 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:23.724 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:01:23.724 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:23.724 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:01:23.725 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:23.725 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:23.725 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:01:23.725 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:23.725 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:01:23.726 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:23.726 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:01:23.726 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:23.727 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:23.727 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:23.727 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:23.727 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:01:23.728 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:23.728 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:23.728 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:01:23.728 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:23.728 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:23.729 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:23.729 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:01:23.729 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:23.729 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:23.730 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:23.730 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:23.730 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:01:23.730 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:23.731 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:23.731 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T20:01:23.731 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:23.732 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:23.732 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:23.732 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T20:01:23.732 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:23.733 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:23.733 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:23.734 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:23.734 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:01:23.734 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:23.734 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:23.734 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:01:23.735 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:23.735 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:23.735 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:23.736 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:01:23.736 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:23.736 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:23.737 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:23.737 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:01:23.738 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:23.738 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:01:23.739 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:01:23.739 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:01:23.740 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:23.740 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:01:23.742 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.543574_190.1 2021-09-16T20:01:28.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:28.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:28.744 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:01:28.787 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:01:28.788 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:01:28.839 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:01:28.840 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:01:28.840 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:01:28.840 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:01:28.841 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:01:28.841 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:01:28.841 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:28.841 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:28.842 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:01:28.842 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:28.842 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:01:28.843 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:28.843 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:28.843 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:01:28.843 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:28.844 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:01:28.844 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:28.844 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:01:28.844 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:28.844 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:28.845 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:28.845 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:28.845 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:01:28.845 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:28.845 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:28.846 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:01:28.846 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:28.846 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:28.846 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:28.847 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:01:28.847 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:28.847 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:28.847 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:28.848 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:28.848 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:01:28.848 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:28.848 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:28.849 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T20:01:28.849 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:28.849 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:28.849 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:28.850 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T20:01:28.850 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:28.850 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:28.850 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:28.851 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:28.851 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:01:28.851 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:28.851 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:28.851 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:01:28.852 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:28.852 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:28.852 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:28.852 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:01:28.853 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:28.853 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:28.853 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:28.853 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:01:28.853 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:28.854 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:01:28.854 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:01:28.854 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:01:28.854 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:28.855 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:01:28.856 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.488881_181.1 2021-09-16T20:01:33.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:33.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:33.858 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:01:33.902 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:01:33.903 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:01:33.966 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:01:33.966 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:01:33.966 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:01:33.967 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:01:33.967 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:01:33.967 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:01:33.967 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:33.968 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:33.968 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:01:33.968 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:33.968 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:01:33.969 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:33.969 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:33.969 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:01:33.969 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:33.970 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:01:33.970 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:33.970 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:01:33.970 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:33.971 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:33.971 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:33.971 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:33.971 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:01:33.972 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:33.972 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:33.972 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:01:33.972 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:33.972 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:33.973 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:33.973 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:01:33.973 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:33.973 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:33.974 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:33.974 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:33.974 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:01:33.974 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:33.975 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:33.975 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T20:01:33.975 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:33.975 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:33.975 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:33.976 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T20:01:33.976 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:33.976 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:33.976 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:33.977 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:33.977 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:01:33.977 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:33.977 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:33.978 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:01:33.978 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:33.978 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:33.978 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:33.978 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:01:33.979 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:33.979 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:33.979 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:33.979 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:01:33.980 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:33.980 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:01:33.980 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:01:33.980 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:01:33.980 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:33.981 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:01:33.982 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.489753_182.1 2021-09-16T20:01:38.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:38.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:38.987 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:01:39.030 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:01:39.030 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:01:39.103 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:01:39.103 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:01:39.103 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:01:39.104 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:01:39.104 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:01:39.104 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:01:39.104 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:39.105 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:39.105 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:01:39.105 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:39.105 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:01:39.105 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:39.106 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:39.106 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:01:39.106 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:39.107 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:01:39.107 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:39.107 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:01:39.107 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:39.107 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:39.108 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:39.108 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:39.108 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:01:39.108 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:39.109 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:39.109 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:01:39.109 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:39.109 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:39.109 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:39.110 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:01:39.110 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:39.110 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:39.110 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:39.111 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:39.111 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:01:39.111 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:39.111 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:39.111 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.725649_200.1", 2021-09-16T20:01:39.112 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:39.112 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:39.112 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:39.112 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.725649Z", 2021-09-16T20:01:39.112 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:39.113 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:39.113 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:39.113 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:39.113 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:01:39.114 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:39.114 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:39.114 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:01:39.114 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:39.114 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:39.115 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:39.115 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:01:39.115 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:39.115 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:39.116 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:39.116 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:01:39.116 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:39.116 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:01:39.117 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:01:39.117 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:01:39.117 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:39.117 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:01:39.119 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.725649_200.1 2021-09-16T20:01:44.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:44.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:44.122 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:01:44.164 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:01:44.165 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:01:44.227 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:01:44.228 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:01:44.228 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:01:44.228 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:01:44.229 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:01:44.229 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:01:44.229 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:44.229 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:44.230 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:01:44.230 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:44.230 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:01:44.230 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:44.230 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:44.231 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:01:44.231 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:44.231 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:01:44.231 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:44.232 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:01:44.232 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:44.232 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:44.232 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:44.233 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:44.233 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:01:44.233 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:44.233 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:44.233 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:01:44.234 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:44.234 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:44.234 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:44.234 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:01:44.234 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:44.235 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:44.235 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:44.235 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:44.235 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:01:44.236 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:44.236 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:44.236 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T20:01:44.236 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:44.236 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:44.237 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:44.237 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T20:01:44.237 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:44.237 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:44.238 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:44.238 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:44.238 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:01:44.238 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:44.238 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:44.239 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:01:44.239 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:44.239 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:44.239 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:44.240 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:01:44.240 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:44.240 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:44.240 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:44.240 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:01:44.241 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:44.241 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:01:44.241 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:01:44.241 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:01:44.242 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:44.242 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:01:44.243 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.488881_181.1 2021-09-16T20:01:49.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:49.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:49.245 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:01:49.289 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:01:49.290 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:01:49.353 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:01:49.354 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:01:49.354 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:01:49.354 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:01:49.354 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:01:49.355 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:01:49.355 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:49.356 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:49.356 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:01:49.356 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:49.356 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:01:49.357 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:49.357 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:49.357 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:01:49.357 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:49.358 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:01:49.358 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:49.358 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:01:49.358 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:49.359 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:49.359 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:49.359 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:49.360 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:01:49.360 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:49.360 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:49.360 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:01:49.361 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:49.361 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:49.361 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:49.361 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:01:49.361 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:49.362 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:49.362 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:49.362 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:49.362 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:01:49.363 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:49.363 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:49.363 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T20:01:49.363 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:49.363 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:49.364 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:49.364 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T20:01:49.364 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:49.365 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:49.365 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:49.365 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:49.366 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:01:49.366 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:49.366 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:49.366 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:01:49.367 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:49.367 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:49.367 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:49.367 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:01:49.368 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:49.368 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:49.368 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:49.368 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:01:49.369 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:49.369 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:01:49.369 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:01:49.369 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:01:49.370 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:49.370 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:01:49.371 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.488881_181.1 2021-09-16T20:01:54.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:54.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:54.374 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:01:54.417 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:01:54.417 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:01:54.497 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:01:54.498 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:01:54.498 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:01:54.498 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:01:54.499 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:01:54.499 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:01:54.499 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:54.499 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:54.500 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:01:54.500 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:54.500 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:01:54.500 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:54.501 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:54.501 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:01:54.501 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:54.501 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:01:54.501 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:54.502 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:01:54.502 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:54.502 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:54.502 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:54.503 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:54.503 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:01:54.503 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:54.503 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:54.504 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:01:54.504 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:54.504 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:54.504 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:54.505 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:01:54.505 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:54.505 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:54.505 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:54.505 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:54.506 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:01:54.506 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:54.506 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:54.506 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T20:01:54.507 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:54.507 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:54.507 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:54.507 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T20:01:54.508 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:54.508 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:54.508 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:54.508 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:54.508 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:01:54.509 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:54.509 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:54.509 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:01:54.509 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:54.510 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:54.510 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:54.510 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:01:54.510 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:54.510 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:54.511 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:54.511 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:01:54.511 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:54.511 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:01:54.512 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:01:54.512 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:01:54.512 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:54.512 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:01:54.513 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.489753_182.1 2021-09-16T20:01:59.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:59.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:01:59.516 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:01:59.565 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:01:59.566 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:01:59.631 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:01:59.632 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:01:59.633 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:01:59.633 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:01:59.634 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:01:59.634 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:01:59.634 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:59.634 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:59.635 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:01:59.635 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:59.635 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:01:59.635 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:59.636 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:59.636 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:01:59.636 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:59.636 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:01:59.637 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:59.637 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:01:59.637 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:59.637 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:59.637 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:59.638 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:59.638 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:01:59.638 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:59.639 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:59.639 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:01:59.639 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:59.639 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:59.639 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:59.640 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:01:59.640 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:59.640 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:59.640 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:59.641 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:59.641 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:01:59.641 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:59.641 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:59.642 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T20:01:59.642 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:59.642 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:59.642 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:59.643 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T20:01:59.643 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:59.643 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:59.643 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:59.644 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:01:59.644 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:01:59.644 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:01:59.645 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:01:59.645 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:01:59.645 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:01:59.646 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:01:59.646 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:01:59.646 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:01:59.646 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:01:59.646 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:59.647 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:59.647 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:01:59.647 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:01:59.647 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:01:59.648 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:01:59.648 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:01:59.648 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:01:59.648 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:01:59.650 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.489753_182.1 2021-09-16T20:02:04.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:04.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:04.652 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:02:04.697 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:02:04.697 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:02:04.764 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:02:04.764 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:02:04.764 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:02:04.765 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:02:04.765 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:02:04.765 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:02:04.765 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:04.766 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:04.766 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:02:04.766 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:04.766 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:02:04.767 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:04.767 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:04.767 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:02:04.767 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:04.767 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:02:04.768 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:04.768 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:02:04.768 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:04.768 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:04.769 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:04.769 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:04.769 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:02:04.769 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:04.769 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:04.770 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:02:04.770 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:04.770 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:04.770 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:04.771 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:02:04.771 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:04.771 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:04.771 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:04.771 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:04.772 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:02:04.772 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:04.772 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:04.772 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.517759_187.1", 2021-09-16T20:02:04.773 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:04.773 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:04.773 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:04.773 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.517759Z", 2021-09-16T20:02:04.773 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:04.774 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:04.774 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:04.774 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:04.774 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:02:04.775 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:04.775 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:04.775 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:02:04.775 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:04.775 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:04.776 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:04.776 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:02:04.776 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:04.776 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:04.777 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:04.777 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:02:04.777 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:04.777 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:02:04.777 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:02:04.778 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:02:04.778 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:04.778 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:02:04.779 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.517759_187.1 2021-09-16T20:02:09.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:09.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:09.782 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:02:09.830 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:02:09.831 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:02:09.902 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:02:09.903 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:02:09.903 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:02:09.903 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:02:09.903 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:02:09.904 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:02:09.904 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:09.904 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:09.904 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:02:09.905 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:09.905 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:02:09.905 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:09.905 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:09.905 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:02:09.906 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:09.906 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:02:09.906 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:09.906 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:02:09.907 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:09.907 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:09.907 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:09.907 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:09.907 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:02:09.908 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:09.908 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:09.908 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:02:09.908 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:09.909 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:09.909 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:09.909 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:02:09.909 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:09.909 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:09.910 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:09.910 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:09.910 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:02:09.910 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:09.910 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:09.911 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.668185_197.1", 2021-09-16T20:02:09.911 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:09.911 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:09.911 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:09.912 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.668185Z", 2021-09-16T20:02:09.912 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:09.912 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:09.912 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:09.912 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:09.913 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:02:09.913 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:09.913 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:09.913 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:02:09.913 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:09.914 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:09.914 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:09.914 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:02:09.914 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:09.915 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:09.915 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:09.915 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:02:09.915 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:09.915 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:02:09.916 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:02:09.916 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:02:09.916 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:09.916 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:02:09.918 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.668185_197.1 2021-09-16T20:02:14.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:14.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:14.920 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:02:14.968 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:02:14.969 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:02:15.044 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:02:15.044 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:02:15.044 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:02:15.045 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:02:15.045 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:02:15.045 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:02:15.045 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:15.046 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:15.046 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:02:15.046 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:15.046 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:02:15.047 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:15.047 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:15.047 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:02:15.047 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:15.047 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:02:15.048 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:15.048 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:02:15.048 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:15.048 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:15.049 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:15.049 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:15.049 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:02:15.049 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:15.049 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:15.050 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:02:15.050 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:15.050 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:15.050 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:15.051 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:02:15.051 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:15.051 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:15.051 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:15.051 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:15.052 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:02:15.052 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:15.052 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:15.052 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.538756_188.1", 2021-09-16T20:02:15.053 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:15.053 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:15.053 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:15.053 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.538756Z", 2021-09-16T20:02:15.053 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:15.054 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:15.054 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:15.054 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:15.054 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:02:15.055 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:15.055 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:15.055 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:02:15.055 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:15.055 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:15.056 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:15.056 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:02:15.056 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:15.056 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:15.057 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:15.057 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:02:15.057 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:15.057 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:02:15.057 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:02:15.058 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:02:15.058 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:15.058 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:02:15.059 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.538756_188.1 2021-09-16T20:02:20.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:20.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:20.062 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:02:20.104 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:02:20.105 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:02:20.156 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:02:20.156 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:02:20.156 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:02:20.157 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:02:20.157 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:02:20.157 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:02:20.157 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:20.158 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:20.158 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:02:20.158 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:20.158 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:02:20.159 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:20.159 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:20.159 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:02:20.159 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:20.159 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:02:20.160 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:20.160 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:02:20.160 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:20.160 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:20.161 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:20.161 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:20.161 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:02:20.161 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:20.161 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:20.162 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:02:20.162 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:20.162 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:20.162 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:20.163 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:02:20.163 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:20.163 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:20.163 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:20.163 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:20.164 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:02:20.164 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:20.164 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:20.164 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.517759_187.1", 2021-09-16T20:02:20.164 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:20.165 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:20.165 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:20.165 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.517759Z", 2021-09-16T20:02:20.165 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:20.166 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:20.166 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:20.166 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:20.166 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:02:20.166 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:20.167 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:20.167 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:02:20.167 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:20.167 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:20.167 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:20.168 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:02:20.168 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:20.168 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:20.168 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:20.169 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:02:20.169 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:20.169 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:02:20.169 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:02:20.169 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:02:20.170 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:20.170 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:02:20.171 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.517759_187.1 2021-09-16T20:02:25.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:25.173 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:25.174 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:02:25.218 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:02:25.219 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:02:25.288 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:02:25.289 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:02:25.289 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:02:25.289 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:02:25.289 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:02:25.290 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:02:25.290 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:25.290 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:25.290 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:02:25.291 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:25.291 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:02:25.291 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:25.291 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:25.292 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:02:25.292 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:25.292 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:02:25.292 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:25.292 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:02:25.293 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:25.293 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:25.293 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:25.293 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:25.293 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:02:25.294 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:25.294 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:25.294 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:02:25.294 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:25.295 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:25.295 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:25.295 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:02:25.295 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:25.295 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:25.296 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:25.296 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:25.296 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:02:25.296 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:25.296 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:25.297 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.742140_204.1", 2021-09-16T20:02:25.297 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:25.298 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:25.298 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:25.299 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.742140Z", 2021-09-16T20:02:25.300 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:25.300 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:25.300 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:25.300 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:25.301 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:02:25.301 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:25.301 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:25.301 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:02:25.301 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:25.302 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:25.302 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:25.302 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:02:25.302 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:25.303 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:25.303 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:25.303 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:02:25.303 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:25.303 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:02:25.304 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:02:25.304 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:02:25.304 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:25.304 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:02:25.306 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.742140_204.1 2021-09-16T20:02:30.308 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:30.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:30.309 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:02:30.351 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:02:30.352 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:02:30.403 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:02:30.403 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:02:30.404 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:02:30.404 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:02:30.404 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:02:30.404 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:02:30.404 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:30.405 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:30.405 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:02:30.405 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:30.405 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:02:30.406 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:30.406 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:30.406 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:02:30.406 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:30.407 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:02:30.407 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:30.407 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:02:30.407 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:30.407 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:30.408 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:30.408 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:30.408 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:02:30.408 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:30.409 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:30.409 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:02:30.409 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:30.409 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:30.409 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:30.410 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:02:30.410 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:30.410 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:30.410 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:30.411 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:30.411 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:02:30.411 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:30.411 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:30.412 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.517759_187.1", 2021-09-16T20:02:30.412 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:30.412 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:30.412 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:30.412 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.517759Z", 2021-09-16T20:02:30.413 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:30.413 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:30.413 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:30.413 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:30.414 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:02:30.414 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:30.414 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:30.414 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:02:30.414 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:30.415 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:30.415 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:30.415 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:02:30.415 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:30.415 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:30.416 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:30.416 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:02:30.416 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:30.417 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:02:30.417 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:02:30.417 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:02:30.417 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:30.417 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:02:30.419 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.517759_187.1 2021-09-16T20:02:35.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:35.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:35.422 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:02:35.466 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:02:35.467 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:02:35.540 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:02:35.540 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:02:35.540 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:02:35.541 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:02:35.541 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:02:35.541 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:02:35.541 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:35.541 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:35.542 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:02:35.542 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:35.542 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:02:35.542 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:35.543 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:35.543 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:02:35.543 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:35.543 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:02:35.543 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:35.544 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:02:35.544 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:35.544 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:35.544 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:35.545 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:35.545 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:02:35.545 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:35.545 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:35.545 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:02:35.546 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:35.546 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:35.546 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:35.546 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:02:35.547 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:35.547 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:35.547 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:35.547 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:35.548 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:02:35.548 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:35.548 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:35.548 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.511559_185.1", 2021-09-16T20:02:35.549 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:35.549 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:35.549 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:35.549 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.511559Z", 2021-09-16T20:02:35.549 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:35.550 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:35.550 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:35.550 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:35.550 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:02:35.551 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:35.551 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:35.551 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:02:35.551 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:35.552 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:35.552 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:35.552 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:02:35.552 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:35.552 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:35.553 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:35.553 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:02:35.553 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:35.553 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:02:35.554 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:02:35.554 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:02:35.554 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:35.554 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:02:35.555 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.511559_185.1 2021-09-16T20:02:40.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:40.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:40.558 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:02:40.600 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:02:40.601 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:02:40.654 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:02:40.655 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:02:40.655 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:02:40.655 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:02:40.656 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:02:40.656 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:02:40.656 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:40.656 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:40.656 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:02:40.657 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:40.657 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:02:40.657 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:40.657 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:40.658 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:02:40.658 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:40.658 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:02:40.658 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:40.659 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:02:40.659 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:40.659 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:40.659 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:40.660 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:40.660 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:02:40.661 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:40.661 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:40.661 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:02:40.661 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:40.662 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:40.662 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:40.662 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:02:40.662 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:40.662 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:40.663 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:40.663 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:40.663 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:02:40.663 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:40.664 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:40.664 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T20:02:40.664 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:40.664 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:40.664 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:40.665 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T20:02:40.665 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:40.666 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:40.666 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:40.666 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:40.666 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:02:40.666 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:40.667 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:40.667 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:02:40.667 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:40.667 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:40.668 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:40.668 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:02:40.668 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:40.668 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:40.668 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:40.669 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:02:40.669 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:40.669 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:02:40.669 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:02:40.669 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:02:40.670 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:40.670 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:02:40.671 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.488881_181.1 2021-09-16T20:02:45.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:45.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:45.674 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:02:45.724 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:02:45.725 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:02:45.785 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:02:45.785 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:02:45.785 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:02:45.785 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:02:45.786 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:02:45.786 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:02:45.786 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:45.786 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:45.786 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:02:45.787 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:45.787 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:02:45.787 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:45.787 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:45.788 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:02:45.788 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:45.788 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:02:45.788 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:45.788 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:02:45.789 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:45.789 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:45.789 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:45.789 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:45.789 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:02:45.790 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:45.790 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:45.790 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:02:45.790 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:45.791 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:45.791 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:45.791 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:02:45.791 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:45.791 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:45.792 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:45.792 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:45.792 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:02:45.792 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:45.792 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:45.793 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.492123_183.1", 2021-09-16T20:02:45.793 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:45.793 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:45.793 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:45.793 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.492123Z", 2021-09-16T20:02:45.794 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:45.794 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:45.794 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:45.794 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:45.795 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:02:45.795 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:45.795 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:45.795 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:02:45.795 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:45.796 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:45.796 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:45.796 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:02:45.796 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:45.796 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:45.797 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:45.797 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:02:45.797 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:45.797 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:02:45.798 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:02:45.798 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:02:45.798 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:45.798 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:02:45.799 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.492123_183.1 2021-09-16T20:02:50.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:50.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:50.802 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:02:50.863 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:02:50.863 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:02:50.960 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:02:50.961 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:02:50.961 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:02:50.961 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:02:50.961 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:02:50.962 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:02:50.962 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:50.962 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:50.962 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:02:50.963 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:50.963 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:02:50.963 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:50.963 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:50.963 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:02:50.964 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:50.964 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:02:50.964 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:50.965 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:02:50.965 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:50.965 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:50.965 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:50.965 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:50.966 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:02:50.966 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:50.967 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:50.967 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:02:50.967 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:50.967 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:50.967 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:50.968 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:02:50.968 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:50.968 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:50.968 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:50.969 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:50.969 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:02:50.969 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:50.969 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:50.970 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.665525_196.1", 2021-09-16T20:02:50.970 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:50.971 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:50.971 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:50.971 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.665525Z", 2021-09-16T20:02:50.972 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:50.972 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:50.972 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:50.972 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:50.972 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:02:50.973 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:50.973 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:50.973 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:02:50.973 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:50.973 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:50.974 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:50.974 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:02:50.974 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:50.974 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:50.974 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:50.975 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:02:50.975 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:50.975 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:02:50.975 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:02:50.976 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:02:50.976 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:50.976 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:02:50.978 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.665525_196.1 2021-09-16T20:02:55.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:55.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:02:55.981 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:02:56.065 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:02:56.065 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:02:56.212 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:02:56.212 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:02:56.213 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:02:56.213 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:02:56.213 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:02:56.213 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:02:56.213 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:56.214 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:56.214 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:02:56.214 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:56.214 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:02:56.215 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:56.215 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:56.215 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:02:56.215 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:56.215 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:02:56.216 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:56.216 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:02:56.216 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:56.216 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:56.217 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:56.217 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:56.217 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:02:56.217 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:56.217 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:56.218 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:02:56.218 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:56.218 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:56.218 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:56.219 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:02:56.219 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:56.219 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:56.219 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:56.219 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:56.220 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:02:56.220 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:56.220 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:56.220 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T20:02:56.220 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:56.221 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:56.221 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:56.221 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T20:02:56.221 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:56.221 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:56.222 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:56.222 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:02:56.222 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:02:56.222 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:02:56.223 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:02:56.223 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:02:56.223 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:02:56.223 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:02:56.223 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:02:56.224 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:02:56.224 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:02:56.224 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:56.224 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:56.225 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:02:56.225 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:02:56.225 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:02:56.225 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:02:56.225 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:02:56.226 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:02:56.226 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:02:56.227 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.513492_186.1 2021-09-16T20:03:01.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:01.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:01.230 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:03:01.303 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:03:01.304 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:03:01.448 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:03:01.448 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:03:01.448 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:03:01.449 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:03:01.449 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:03:01.449 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:03:01.449 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:01.449 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:01.450 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:03:01.450 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:01.450 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:03:01.450 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:01.451 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:01.451 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:03:01.451 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:01.451 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:03:01.452 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:01.452 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:03:01.452 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:01.452 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:01.453 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:01.453 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:01.453 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:03:01.453 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:01.453 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:01.454 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:03:01.454 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:01.454 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:01.454 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:01.455 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:03:01.455 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:01.455 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:01.455 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:01.455 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:01.456 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:03:01.456 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:01.456 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:01.457 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.488881_181.1", 2021-09-16T20:03:01.457 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:01.457 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:01.457 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:01.457 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.488881Z", 2021-09-16T20:03:01.458 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:01.458 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:01.458 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:01.458 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:01.459 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:03:01.459 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:01.459 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:01.459 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:03:01.459 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:01.460 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:01.460 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:01.460 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:03:01.460 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:01.461 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:01.461 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:01.461 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:03:01.461 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:01.461 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:03:01.462 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:03:01.462 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:03:01.462 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:01.462 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:03:01.464 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.488881_181.1 2021-09-16T20:03:06.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:06.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:06.466 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:03:06.509 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:03:06.509 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:03:06.576 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:03:06.576 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:03:06.577 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:03:06.577 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:03:06.577 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:03:06.577 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:03:06.577 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:06.578 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:06.578 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:03:06.578 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:06.578 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:03:06.579 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:06.579 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:06.579 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:03:06.579 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:06.579 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:03:06.580 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:06.580 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:03:06.580 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:06.580 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:06.580 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:06.581 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:06.581 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:03:06.581 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:06.581 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:06.581 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:03:06.582 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:06.582 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:06.582 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:06.582 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:03:06.583 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:06.583 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:06.583 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:06.583 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:06.583 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:03:06.584 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:06.584 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:06.584 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T20:03:06.584 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:06.585 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:06.585 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:06.585 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T20:03:06.585 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:06.585 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:06.586 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:06.586 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:06.586 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:03:06.586 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:06.586 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:06.587 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:03:06.587 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:06.587 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:06.587 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:06.588 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:03:06.588 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:06.588 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:06.588 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:06.588 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:03:06.589 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:06.589 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:03:06.589 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:03:06.589 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:03:06.589 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:06.590 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:03:06.591 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.489753_182.1 2021-09-16T20:03:11.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:11.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:11.593 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:03:11.651 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:03:11.651 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:03:11.722 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:03:11.723 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:03:11.723 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:03:11.723 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:03:11.723 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:03:11.724 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:03:11.724 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:11.724 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:11.724 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:03:11.725 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:11.725 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:03:11.725 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:11.725 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:11.725 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:03:11.726 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:11.726 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:03:11.726 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:11.726 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:03:11.726 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:11.727 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:11.727 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:11.727 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:11.727 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:03:11.727 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:11.728 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:11.728 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:03:11.728 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:11.728 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:11.728 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:11.729 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:03:11.729 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:11.729 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:11.729 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:11.729 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:11.730 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:03:11.730 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:11.730 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:11.730 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821519.105698_224.1", 2021-09-16T20:03:11.730 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:11.731 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:11.731 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:11.731 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:19.105698Z", 2021-09-16T20:03:11.731 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:11.731 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:11.732 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:11.732 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:11.732 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:03:11.732 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:11.732 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:11.733 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:03:11.733 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:11.733 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:11.733 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:11.734 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:03:11.734 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:11.734 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:11.734 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:11.734 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:03:11.735 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:11.735 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:03:11.735 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:03:11.735 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:03:11.735 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:11.736 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:03:11.737 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821519.105698_224.1 2021-09-16T20:03:16.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:16.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:16.739 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:03:16.822 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:03:16.822 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:03:16.950 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:03:16.951 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:03:16.951 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:03:16.951 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:03:16.951 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:03:16.952 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:03:16.952 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:16.952 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:16.952 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:03:16.953 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:16.953 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:03:16.953 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:16.953 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:16.954 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:03:16.954 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:16.954 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:03:16.954 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:16.954 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:03:16.955 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:16.955 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:16.955 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:16.955 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:16.956 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:03:16.956 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:16.956 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:16.956 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:03:16.957 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:16.957 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:16.957 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:16.957 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:03:16.957 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:16.958 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:16.958 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:16.958 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:16.958 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:03:16.959 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:16.959 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:16.959 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T20:03:16.959 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:16.960 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:16.960 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:16.960 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T20:03:16.960 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:16.960 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:16.961 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:16.961 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:16.961 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:03:16.961 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:16.961 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:16.962 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:03:16.962 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:16.962 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:16.962 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:16.963 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:03:16.963 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:16.963 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:16.963 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:16.964 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:03:16.964 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:16.964 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:03:16.964 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:03:16.964 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:03:16.965 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:16.965 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:03:16.965 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.513492_186.1 2021-09-16T20:03:21.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:21.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:21.968 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:03:22.050 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:03:22.050 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:03:22.230 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:03:22.230 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:03:22.230 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:03:22.230 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:03:22.231 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:03:22.231 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:03:22.231 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:22.231 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:22.232 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:03:22.232 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:22.232 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:03:22.232 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:22.232 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:22.233 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:03:22.233 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:22.233 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:03:22.233 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:22.234 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:03:22.234 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:22.234 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:22.234 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:22.235 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:22.235 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:03:22.235 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:22.235 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:22.236 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:03:22.236 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:22.236 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:22.236 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:22.236 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:03:22.237 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:22.237 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:22.237 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:22.237 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:22.238 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:03:22.238 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:22.238 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:22.238 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T20:03:22.238 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:22.239 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:22.239 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:22.239 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T20:03:22.239 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:22.239 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:22.240 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:22.240 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:22.240 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:03:22.240 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:22.241 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:22.241 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:03:22.241 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:22.241 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:22.241 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:22.242 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:03:22.242 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:22.242 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:22.242 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:22.243 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:03:22.243 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:22.243 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:03:22.243 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:03:22.243 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:03:22.244 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:22.244 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:03:22.245 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.489753_182.1 2021-09-16T20:03:27.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:27.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:27.248 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:03:27.334 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:03:27.334 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:03:27.535 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:03:27.536 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:03:27.536 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:03:27.536 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:03:27.537 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:03:27.537 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:03:27.537 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:27.537 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:27.537 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:03:27.538 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:27.538 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:03:27.538 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:27.538 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:27.539 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:03:27.539 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:27.539 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:03:27.539 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:27.539 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:03:27.540 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:27.540 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:27.540 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:27.540 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:27.541 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:03:27.541 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:27.541 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:27.541 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:03:27.541 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:27.542 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:27.542 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:27.542 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:03:27.542 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:27.542 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:27.543 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:27.543 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:27.543 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:03:27.543 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:27.544 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:27.544 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.517759_187.1", 2021-09-16T20:03:27.544 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:27.544 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:27.545 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:27.545 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.517759Z", 2021-09-16T20:03:27.545 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:27.545 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:27.545 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:27.546 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:27.546 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:03:27.546 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:27.546 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:27.547 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:03:27.547 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:27.547 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:27.547 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:27.547 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:03:27.548 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:27.548 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:27.548 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:27.548 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:03:27.548 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:27.549 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:03:27.549 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:03:27.549 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:03:27.549 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:27.550 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:03:27.551 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.517759_187.1 2021-09-16T20:03:32.552 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:32.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:32.553 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:03:32.643 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:03:32.643 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:03:32.777 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:03:32.777 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:03:32.777 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:03:32.778 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:03:32.778 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:03:32.778 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:03:32.778 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:32.779 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:32.779 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:03:32.779 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:32.779 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:03:32.779 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:32.780 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:32.780 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:03:32.780 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:32.780 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:03:32.781 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:32.781 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:03:32.781 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:32.781 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:32.781 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:32.782 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:32.782 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:03:32.782 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:32.782 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:32.783 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:03:32.783 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:32.783 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:32.783 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:32.783 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:03:32.784 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:32.784 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:32.784 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:32.784 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:32.785 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:03:32.785 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:32.785 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:32.785 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.665525_196.1", 2021-09-16T20:03:32.785 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:32.786 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:32.786 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:32.786 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.665525Z", 2021-09-16T20:03:32.786 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:32.786 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:32.787 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:32.787 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:32.787 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:03:32.787 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:32.788 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:32.788 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:03:32.788 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:32.788 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:32.788 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:32.789 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:03:32.789 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:32.789 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:32.789 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:32.789 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:03:32.790 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:32.790 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:03:32.790 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:03:32.790 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:03:32.791 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:32.791 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:03:32.792 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.665525_196.1 2021-09-16T20:03:37.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:37.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:37.794 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:03:37.865 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:03:37.865 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:03:37.995 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:03:37.995 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:03:37.996 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:03:37.996 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:03:37.996 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:03:37.996 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:03:37.997 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:37.997 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:37.997 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:03:37.997 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:37.997 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:03:37.998 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:37.998 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:37.998 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:03:37.998 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:37.999 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:03:37.999 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:37.999 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:03:37.999 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:38.000 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:38.000 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:38.000 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:38.000 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:03:38.001 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:38.001 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:38.001 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:03:38.001 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:38.001 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:38.002 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:38.002 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:03:38.002 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:38.002 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:38.002 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:38.003 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:38.003 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:03:38.003 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:38.003 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:38.004 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T20:03:38.004 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:38.004 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:38.004 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:38.004 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T20:03:38.005 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:38.005 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:38.005 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:38.005 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:38.006 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:03:38.006 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:38.006 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:38.006 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:03:38.007 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:38.007 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:38.007 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:38.007 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:03:38.007 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:38.008 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:38.008 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:38.008 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:03:38.008 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:38.009 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:03:38.009 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:03:38.009 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:03:38.009 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:38.009 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:03:38.011 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.513492_186.1 2021-09-16T20:03:43.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:43.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:43.013 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:03:43.091 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:03:43.092 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:03:43.238 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:03:43.238 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:03:43.238 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:03:43.239 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:03:43.239 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:03:43.239 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:03:43.240 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:43.240 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:43.241 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:03:43.241 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:43.241 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:03:43.242 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:43.242 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:43.243 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:03:43.243 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:43.244 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:03:43.244 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:43.245 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:03:43.245 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:43.246 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:43.246 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:43.247 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:43.247 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:03:43.247 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:43.248 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:43.248 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:03:43.249 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:43.249 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:43.249 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:43.249 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:03:43.250 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:43.250 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:43.250 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:43.250 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:43.251 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:03:43.251 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:43.251 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:43.251 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.492123_183.1", 2021-09-16T20:03:43.251 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:43.252 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:43.252 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:43.252 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.492123Z", 2021-09-16T20:03:43.252 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:43.253 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:43.253 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:43.253 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:43.253 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:03:43.254 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:43.254 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:43.254 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:03:43.254 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:43.255 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:43.255 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:43.255 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:03:43.255 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:43.256 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:43.256 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:43.256 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:03:43.257 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:43.257 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:03:43.257 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:03:43.257 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:03:43.258 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:43.258 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:03:43.259 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.492123_183.1 2021-09-16T20:03:48.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:48.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:48.262 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:03:48.360 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:03:48.361 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:03:48.651 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:03:48.652 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:03:48.652 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:03:48.652 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:03:48.652 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:03:48.653 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:03:48.653 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:48.653 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:48.653 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:03:48.654 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:48.654 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:03:48.654 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:48.654 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:48.655 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:03:48.655 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:48.655 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:03:48.655 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:48.655 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:03:48.656 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:48.656 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:48.656 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:48.656 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:48.656 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:03:48.657 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:48.657 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:48.657 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:03:48.657 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:48.657 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:48.658 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:48.658 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:03:48.658 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:48.658 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:48.659 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:48.659 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:48.659 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:03:48.659 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:48.659 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:48.660 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.511559_185.1", 2021-09-16T20:03:48.660 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:48.660 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:48.660 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:48.661 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.511559Z", 2021-09-16T20:03:48.661 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:48.661 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:48.661 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:48.661 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:48.662 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:03:48.662 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:48.662 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:48.662 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:03:48.663 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:48.663 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:48.663 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:48.663 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:03:48.664 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:48.664 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:48.664 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:48.664 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:03:48.664 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:48.665 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:03:48.665 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:03:48.665 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:03:48.665 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:48.665 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:03:48.667 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.511559_185.1 2021-09-16T20:03:53.668 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:53.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:53.670 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:03:53.743 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:03:53.743 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:03:53.976 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:03:53.976 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:03:53.976 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:03:53.977 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:03:53.977 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:03:53.977 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:03:53.977 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:53.978 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:53.978 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:03:53.978 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:53.978 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:03:53.978 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:53.979 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:53.979 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:03:53.979 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:53.979 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:03:53.980 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:53.980 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:03:53.980 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:53.980 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:53.980 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:53.981 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:53.981 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:03:53.981 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:53.981 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:53.982 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:03:53.982 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:53.982 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:53.982 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:53.983 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:03:53.983 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:53.983 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:53.983 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:53.984 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:53.984 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:03:53.984 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:53.984 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:53.985 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T20:03:53.985 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:53.985 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:53.985 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:53.986 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T20:03:53.986 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:53.986 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:53.986 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:53.987 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:53.987 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:03:53.987 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:53.987 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:53.987 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:03:53.988 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:53.988 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:53.988 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:53.988 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:03:53.989 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:53.989 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:53.989 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:53.989 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:03:53.989 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:53.990 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:03:53.990 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:03:53.990 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:03:53.990 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:53.991 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:03:53.992 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.489753_182.1 2021-09-16T20:03:58.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:58.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:03:58.995 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:03:59.069 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:03:59.070 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:03:59.150 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:03:59.150 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:03:59.150 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:03:59.151 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:03:59.151 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:03:59.151 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:03:59.151 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:59.152 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:59.152 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:03:59.152 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:59.152 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:03:59.152 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:59.153 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:59.153 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:03:59.153 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:59.154 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:03:59.154 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:59.154 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:03:59.154 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:59.155 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:59.155 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:59.155 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:59.155 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:03:59.155 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:59.156 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:59.156 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:03:59.156 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:59.156 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:59.156 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:59.157 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:03:59.157 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:59.157 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:59.157 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:59.158 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:59.158 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:03:59.158 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:59.158 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:59.158 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T20:03:59.159 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:59.159 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:59.159 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:59.159 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T20:03:59.160 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:59.160 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:59.160 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:59.160 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:03:59.160 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:03:59.161 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:03:59.161 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:03:59.161 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:03:59.161 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:03:59.162 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:03:59.162 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:03:59.162 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:03:59.162 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:03:59.162 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:59.163 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:59.163 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:03:59.163 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:03:59.163 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:03:59.164 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:03:59.164 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:03:59.164 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:03:59.164 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:03:59.165 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.513492_186.1 2021-09-16T20:04:03.154 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T20:04:03.153+0000 7f2fcaf3e700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-16T20:04:03.213 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T20:04:03.214+0000 7f2fcaf3e700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-16T20:04:04.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:04.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:04.169 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:04:04.211 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:04:04.212 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:04:04.263 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:04:04.263 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:04:04.264 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:04:04.264 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:04:04.264 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:04:04.264 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:04:04.264 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:04.265 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:04.265 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:04:04.265 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:04.265 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:04:04.266 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:04.266 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:04.266 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:04:04.266 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:04.267 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:04:04.267 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:04.267 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:04:04.267 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:04.267 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:04.268 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:04.268 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:04.269 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:04:04.269 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:04.269 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:04.269 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:04:04.269 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:04.270 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:04.270 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:04.270 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:04:04.270 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:04.271 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:04.271 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:04.271 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:04.271 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:04:04.272 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:04.272 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:04.272 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.563375_192.1", 2021-09-16T20:04:04.272 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:04.273 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:04.273 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:04.273 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.563375Z", 2021-09-16T20:04:04.273 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:04.274 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:04.274 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:04.274 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:04.274 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:04:04.275 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:04.275 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:04.275 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:04:04.275 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:04.276 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:04.276 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:04.276 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:04:04.277 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:04.277 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:04.277 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:04.277 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:04:04.278 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:04.278 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:04:04.278 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:04:04.278 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:04:04.279 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:04.279 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:04:04.280 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.563375_192.1 2021-09-16T20:04:09.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:09.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:09.283 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:04:09.327 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:04:09.327 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:04:09.389 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:04:09.389 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:04:09.389 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:04:09.389 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:04:09.390 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:04:09.390 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:04:09.390 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:09.390 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:09.391 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:04:09.391 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:09.391 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:04:09.391 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:09.391 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:09.392 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:04:09.392 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:09.392 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:04:09.392 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:09.393 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:04:09.393 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:09.393 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:09.393 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:09.394 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:09.394 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:04:09.394 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:09.394 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:09.394 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:04:09.395 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:09.395 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:09.395 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:09.395 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:04:09.396 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:09.396 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:09.396 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:09.396 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:09.396 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:04:09.397 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:09.397 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:09.397 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.563375_192.1", 2021-09-16T20:04:09.397 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:09.397 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:09.398 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:09.398 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.563375Z", 2021-09-16T20:04:09.398 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:09.398 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:09.399 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:09.399 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:09.399 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:04:09.399 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:09.399 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:09.400 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:04:09.400 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:09.400 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:09.400 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:09.401 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:04:09.401 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:09.401 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:09.401 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:09.401 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:04:09.402 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:09.402 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:04:09.402 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:04:09.402 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:04:09.402 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:09.403 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:04:09.404 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.563375_192.1 2021-09-16T20:04:14.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:14.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:14.407 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:04:14.449 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:04:14.449 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:04:14.506 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:04:14.506 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:04:14.507 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:04:14.507 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:04:14.507 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:04:14.507 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:04:14.508 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:14.508 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:14.508 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:04:14.508 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:14.508 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:04:14.509 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:14.509 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:14.509 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:04:14.509 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:14.510 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:04:14.510 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:14.510 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:04:14.510 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:14.511 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:14.511 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:14.511 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:14.511 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:04:14.511 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:14.512 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:14.512 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:04:14.512 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:14.512 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:14.513 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:14.513 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:04:14.513 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:14.513 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:14.513 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:14.514 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:14.514 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:04:14.514 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:14.514 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:14.514 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T20:04:14.515 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:14.515 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:14.515 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:14.515 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T20:04:14.516 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:14.516 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:14.516 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:14.516 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:14.516 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:04:14.517 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:14.517 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:14.517 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:04:14.517 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:14.517 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:14.518 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:14.518 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:04:14.518 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:14.518 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:14.519 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:14.519 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:04:14.519 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:14.519 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:04:14.519 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:04:14.520 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:04:14.520 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:14.520 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:04:14.521 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.513492_186.1 2021-09-16T20:04:19.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:19.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:19.524 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:04:19.586 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:04:19.586 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:04:19.688 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:04:19.689 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:04:19.689 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:04:19.689 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:04:19.689 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:04:19.690 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:04:19.690 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:19.691 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:19.691 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:04:19.691 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:19.692 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:04:19.692 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:19.692 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:19.693 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:04:19.693 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:19.693 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:04:19.694 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:19.694 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:04:19.695 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:19.695 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:19.695 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:19.696 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:19.696 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:04:19.697 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:19.697 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:19.698 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:04:19.698 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:19.699 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:19.699 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:19.699 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:04:19.700 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:19.700 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:19.700 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:19.701 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:19.701 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:04:19.702 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:19.702 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:19.702 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.513492_186.1", 2021-09-16T20:04:19.702 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:19.703 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:19.703 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:19.703 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.513492Z", 2021-09-16T20:04:19.703 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:19.704 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:19.704 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:19.704 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:19.704 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:04:19.705 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:19.705 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:19.705 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:04:19.706 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:19.706 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:19.706 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:19.707 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:04:19.707 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:19.707 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:19.707 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:19.708 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:04:19.708 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:19.708 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:04:19.708 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:04:19.708 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:04:19.709 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:19.709 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:04:19.710 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.513492_186.1 2021-09-16T20:04:24.712 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:24.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:24.713 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:04:24.773 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:04:24.774 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:04:24.845 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:04:24.845 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:04:24.845 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:04:24.846 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:04:24.846 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:04:24.846 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:04:24.847 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:24.847 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:24.847 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:04:24.847 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:24.848 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:04:24.848 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:24.848 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:24.848 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:04:24.849 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:24.849 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:04:24.849 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:24.849 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:04:24.849 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:24.850 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:24.850 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:24.850 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:24.850 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:04:24.851 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:24.851 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:24.851 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:04:24.851 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:24.851 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:24.852 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:24.852 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:04:24.852 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:24.853 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:24.853 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:24.853 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:24.853 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:04:24.853 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:24.854 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:24.854 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.563375_192.1", 2021-09-16T20:04:24.854 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:24.855 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:24.856 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:24.856 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.563375Z", 2021-09-16T20:04:24.856 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:24.856 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:24.856 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:24.857 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:24.857 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:04:24.857 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:24.857 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:24.858 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:04:24.858 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:24.858 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:24.858 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:24.858 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:04:24.859 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:24.859 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:24.859 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:24.859 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:04:24.859 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:24.860 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:04:24.860 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:04:24.860 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:04:24.860 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:24.861 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:04:24.862 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.563375_192.1 2021-09-16T20:04:29.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:29.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:29.864 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:04:29.923 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:04:29.923 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:04:29.996 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:04:29.996 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:04:29.997 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:04:29.997 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:04:29.997 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:04:29.997 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:04:29.998 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:29.998 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:29.998 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:04:29.998 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:29.999 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:04:29.999 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:29.999 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:29.999 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:04:30.000 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:30.000 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:04:30.000 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:30.000 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:04:30.000 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:30.001 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:30.001 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:30.001 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:30.001 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:04:30.002 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:30.002 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:30.002 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:04:30.002 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:30.003 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:30.003 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:30.003 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:04:30.003 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:30.003 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:30.004 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:30.004 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:30.004 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:04:30.004 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:30.004 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:30.005 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T20:04:30.005 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:30.005 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:30.005 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:30.006 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T20:04:30.006 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:30.006 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:30.006 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:30.007 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:30.007 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:04:30.007 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:30.008 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:30.008 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:04:30.008 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:30.008 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:30.009 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:30.009 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:04:30.009 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:30.009 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:30.009 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:30.010 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:04:30.010 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:30.010 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:04:30.010 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:04:30.011 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:04:30.011 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:30.011 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:04:30.012 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.489753_182.1 2021-09-16T20:04:35.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:35.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:35.015 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:04:35.103 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:04:35.103 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:04:35.279 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:04:35.279 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:04:35.279 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:04:35.279 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:04:35.280 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:04:35.280 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:04:35.280 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:35.280 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:35.281 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:04:35.281 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:35.281 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:04:35.281 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:35.281 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:35.282 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:04:35.282 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:35.282 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:04:35.282 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:35.283 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:04:35.283 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:35.283 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:35.283 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:35.283 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:35.284 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:04:35.284 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:35.284 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:35.284 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:04:35.285 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:35.285 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:35.285 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:35.285 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:04:35.285 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:35.286 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:35.286 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:35.286 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:35.286 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:04:35.287 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:35.287 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:35.287 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T20:04:35.287 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:35.288 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:35.288 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:35.288 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T20:04:35.288 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:35.289 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:35.289 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:35.290 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:35.290 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:04:35.290 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:35.290 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:35.291 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:04:35.291 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:35.291 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:35.291 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:35.292 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:04:35.292 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:35.292 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:35.292 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:35.292 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:04:35.293 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:35.293 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:04:35.293 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:04:35.293 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:04:35.294 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:35.294 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:04:35.295 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.489753_182.1 2021-09-16T20:04:40.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:40.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:40.298 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:04:40.377 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:04:40.378 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:04:40.523 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:04:40.524 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:04:40.524 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:04:40.524 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:04:40.525 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:04:40.525 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:04:40.525 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:40.525 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:40.526 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:04:40.526 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:40.526 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:04:40.526 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:40.526 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:40.527 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:04:40.527 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:40.527 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:04:40.527 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:40.528 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:04:40.529 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:40.529 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:40.529 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:40.529 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:40.530 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:04:40.530 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:40.530 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:40.530 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:04:40.531 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:40.531 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:40.531 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:40.531 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:04:40.532 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:40.532 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:40.532 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:40.532 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:40.532 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:04:40.533 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:40.533 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:40.533 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.665525_196.1", 2021-09-16T20:04:40.533 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:40.534 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:40.534 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:40.534 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.665525Z", 2021-09-16T20:04:40.534 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:40.534 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:40.535 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:40.535 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:40.535 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:04:40.535 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:40.535 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:40.536 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:04:40.536 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:40.536 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:40.536 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:40.536 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:04:40.537 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:40.537 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:40.537 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:40.537 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:04:40.537 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:40.538 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:04:40.538 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:04:40.538 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:04:40.538 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:40.539 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:04:40.540 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.665525_196.1 2021-09-16T20:04:45.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:45.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:45.542 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:04:45.593 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:04:45.593 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:04:45.663 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:04:45.663 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:04:45.664 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:04:45.664 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:04:45.664 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:04:45.664 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:04:45.665 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:45.665 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:45.665 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:04:45.665 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:45.666 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:04:45.666 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:45.666 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:45.666 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:04:45.667 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:45.667 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:04:45.667 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:45.667 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:04:45.667 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:45.668 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:45.668 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:45.668 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:45.668 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:04:45.669 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:45.669 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:45.669 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:04:45.669 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:45.670 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:45.670 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:45.670 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:04:45.670 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:45.671 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:45.671 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:45.671 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:45.671 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:04:45.672 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:45.672 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:45.672 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.492123_183.1", 2021-09-16T20:04:45.672 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:45.673 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:45.673 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:45.673 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.492123Z", 2021-09-16T20:04:45.673 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:45.673 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:45.674 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:45.674 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:45.674 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:04:45.674 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:45.675 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:45.675 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:04:45.675 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:45.675 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:45.676 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:45.676 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:04:45.676 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:45.676 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:45.677 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:45.677 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:04:45.677 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:45.677 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:04:45.677 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:04:45.678 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:04:45.678 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:45.678 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:04:45.680 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.492123_183.1 2021-09-16T20:04:50.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:50.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:50.682 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:04:50.734 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:04:50.734 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:04:50.822 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:04:50.823 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:04:50.823 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:04:50.823 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:04:50.824 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:04:50.824 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:04:50.824 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:50.824 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:50.825 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:04:50.825 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:50.825 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:04:50.825 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:50.826 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:50.826 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:04:50.826 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:50.826 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:04:50.827 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:50.827 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:04:50.827 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:50.828 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:50.828 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:50.828 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:50.828 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:04:50.829 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:50.829 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:50.829 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:04:50.830 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:50.830 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:50.830 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:50.830 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:04:50.831 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:50.831 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:50.831 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:50.831 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:50.832 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:04:50.832 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:50.832 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:50.832 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.508383_184.1", 2021-09-16T20:04:50.832 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:50.833 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:50.833 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:50.833 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.508383Z", 2021-09-16T20:04:50.833 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:50.834 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:50.834 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:50.834 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:50.834 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:04:50.835 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:50.835 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:50.835 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:04:50.835 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:50.835 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:50.836 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:50.836 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:04:50.836 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:50.836 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:50.837 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:50.837 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:04:50.837 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:50.837 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:04:50.837 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:04:50.838 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:04:50.838 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:50.838 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:04:50.839 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.508383_184.1 2021-09-16T20:04:55.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:55.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:04:55.843 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:04:55.909 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:04:55.910 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:04:55.998 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:04:55.998 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:04:55.998 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:04:55.999 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:04:55.999 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:04:55.999 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:04:55.999 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:56.000 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:56.000 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:04:56.000 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:56.000 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:04:56.001 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:56.001 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:56.001 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:04:56.001 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:56.001 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:04:56.002 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:56.002 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:04:56.002 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:56.002 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:56.003 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:56.003 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:56.003 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:04:56.003 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:56.004 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:56.004 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:04:56.004 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:56.004 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:56.004 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:56.005 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:04:56.005 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:56.005 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:56.005 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:56.006 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:56.006 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:04:56.006 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:56.006 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:56.006 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.489753_182.1", 2021-09-16T20:04:56.007 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:56.007 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:56.007 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:56.007 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.489753Z", 2021-09-16T20:04:56.008 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:56.008 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:56.008 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:56.008 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:04:56.008 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:04:56.009 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:04:56.009 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:04:56.009 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:04:56.009 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:04:56.009 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:04:56.010 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:04:56.010 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:04:56.010 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:04:56.010 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:56.011 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:56.012 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:04:56.012 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:04:56.012 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:04:56.013 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:04:56.013 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:04:56.013 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:04:56.013 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:04:56.014 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.489753_182.1 2021-09-16T20:05:01.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:05:01.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:05:01.018 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:05:01.059 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:05:01.060 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:05:01.120 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:05:01.121 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:05:01.121 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:05:01.121 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:05:01.122 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:05:01.122 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:05:01.122 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:05:01.122 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:05:01.123 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:05:01.123 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:05:01.123 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:05:01.123 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:05:01.124 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:05:01.124 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:05:01.124 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:05:01.124 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:05:01.125 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:05:01.125 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:05:01.125 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:05:01.125 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:05:01.125 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:05:01.126 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:05:01.126 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:05:01.126 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:05:01.126 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:05:01.127 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:05:01.127 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:05:01.127 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:05:01.127 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:05:01.128 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:05:01.128 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:05:01.128 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:05:01.128 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:05:01.128 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:05:01.129 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:05:01.129 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:05:01.129 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:05:01.129 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.543574_190.1", 2021-09-16T20:05:01.130 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:05:01.130 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:05:01.130 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:05:01.130 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.543574Z", 2021-09-16T20:05:01.131 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:05:01.131 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:05:01.131 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:05:01.131 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:05:01.131 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:05:01.132 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:05:01.132 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:05:01.132 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:05:01.132 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:05:01.133 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:05:01.133 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:05:01.133 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:05:01.133 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:05:01.134 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:05:01.134 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:05:01.134 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:05:01.134 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:05:01.134 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:05:01.135 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:05:01.135 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:05:01.135 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:05:01.135 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:05:01.137 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822389.517479_289.1 target=1_1631821518.543574_190.1 2021-09-16T20:05:06.139 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2021-09-16T20:05:06.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:05:06.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:05:06.141 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-16T20:05:06.190 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:05:06.190 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:05:06.214 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T20:05:06.213+0000 7f49e6539380 0 failed reading obj info from .rgw.root:zone_info.6395206f-e97e-464f-b7bd-c10485d78bf5: (2) No such file or directory 2021-09-16T20:05:06.214 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T20:05:06.213+0000 7f49e6539380 0 WARNING: could not read zone params for zone id=6395206f-e97e-464f-b7bd-c10485d78bf5 name=test-zone1 2021-09-16T20:05:06.214 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T20:05:06.213+0000 7f49e6539380 0 failed reading obj info from .rgw.root:zone_info.8df190b8-e36d-4ab9-bca1-30bba5bb5ec5: (2) No such file or directory 2021-09-16T20:05:06.215 INFO:teuthology.orchestra.run.smithi118.stderr:2021-09-16T20:05:06.213+0000 7f49e6539380 0 WARNING: could not read zone params for zone id=8df190b8-e36d-4ab9-bca1-30bba5bb5ec5 name=test-zone3 2021-09-16T20:05:06.218 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T20:05:06.219 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "ceba8a70-b881-4808-8acf-e8ddfa5f6daf", 2021-09-16T20:05:06.219 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zone2", 2021-09-16T20:05:06.219 INFO:teuthology.orchestra.run.smithi118.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-16T20:05:06.220 INFO:teuthology.orchestra.run.smithi118.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-16T20:05:06.220 INFO:teuthology.orchestra.run.smithi118.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-16T20:05:06.220 INFO:teuthology.orchestra.run.smithi118.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-16T20:05:06.220 INFO:teuthology.orchestra.run.smithi118.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-16T20:05:06.220 INFO:teuthology.orchestra.run.smithi118.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-16T20:05:06.221 INFO:teuthology.orchestra.run.smithi118.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-16T20:05:06.221 INFO:teuthology.orchestra.run.smithi118.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-16T20:05:06.221 INFO:teuthology.orchestra.run.smithi118.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-16T20:05:06.222 INFO:teuthology.orchestra.run.smithi118.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-16T20:05:06.222 INFO:teuthology.orchestra.run.smithi118.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-16T20:05:06.222 INFO:teuthology.orchestra.run.smithi118.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-16T20:05:06.222 INFO:teuthology.orchestra.run.smithi118.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-16T20:05:06.222 INFO:teuthology.orchestra.run.smithi118.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-16T20:05:06.223 INFO:teuthology.orchestra.run.smithi118.stdout: "system_key": { 2021-09-16T20:05:06.223 INFO:teuthology.orchestra.run.smithi118.stdout: "access_key": "K0MH58GXSJSVNR3I", 2021-09-16T20:05:06.223 INFO:teuthology.orchestra.run.smithi118.stdout: "secret_key": "93vysIyO7YxIyUlpTN7GyE2EQAEwRODi" 2021-09-16T20:05:06.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:05:06.224 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_pools": [ 2021-09-16T20:05:06.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:05:06.224 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "default-placement", 2021-09-16T20:05:06.224 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:05:06.224 INFO:teuthology.orchestra.run.smithi118.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-16T20:05:06.225 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": { 2021-09-16T20:05:06.225 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD": { 2021-09-16T20:05:06.225 INFO:teuthology.orchestra.run.smithi118.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2021-09-16T20:05:06.225 INFO:teuthology.orchestra.run.smithi118.stdout: "compression_type": "random" 2021-09-16T20:05:06.225 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:05:06.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:05:06.226 INFO:teuthology.orchestra.run.smithi118.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-16T20:05:06.226 INFO:teuthology.orchestra.run.smithi118.stdout: "index_type": 0 2021-09-16T20:05:06.226 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:05:06.227 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:05:06.227 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T20:05:06.227 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T20:05:06.227 INFO:teuthology.orchestra.run.smithi118.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-16T20:05:06.228 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T20:05:06.229 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:05:06.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:05:06.230 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-16T20:05:06.273 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:05:06.273 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:05:06.323 INFO:teuthology.orchestra.run.smithi118.stderr:Sending period to new master zone 6395206f-e97e-464f-b7bd-c10485d78bf5 2021-09-16T20:05:06.350 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T20:05:06.351 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:05:06.351 INFO:teuthology.orchestra.run.smithi118.stdout: "epoch": 5, 2021-09-16T20:05:06.351 INFO:teuthology.orchestra.run.smithi118.stdout: "predecessor_uuid": "6be2ae27-983c-4bba-b75f-41d101865b26", 2021-09-16T20:05:06.351 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": [], 2021-09-16T20:05:06.352 INFO:teuthology.orchestra.run.smithi118.stdout: "period_map": { 2021-09-16T20:05:06.352 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:05:06.352 INFO:teuthology.orchestra.run.smithi118.stdout: "zonegroups": [ 2021-09-16T20:05:06.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:05:06.353 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "test-zonegroup", 2021-09-16T20:05:06.353 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zonegroup", 2021-09-16T20:05:06.353 INFO:teuthology.orchestra.run.smithi118.stdout: "api_name": "", 2021-09-16T20:05:06.353 INFO:teuthology.orchestra.run.smithi118.stdout: "is_master": "true", 2021-09-16T20:05:06.353 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2021-09-16T20:05:06.354 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi109.front.sepia.ceph.com:8000" 2021-09-16T20:05:06.354 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T20:05:06.354 INFO:teuthology.orchestra.run.smithi118.stdout: "hostnames": [], 2021-09-16T20:05:06.354 INFO:teuthology.orchestra.run.smithi118.stdout: "hostnames_s3website": [], 2021-09-16T20:05:06.355 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zone": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T20:05:06.355 INFO:teuthology.orchestra.run.smithi118.stdout: "zones": [ 2021-09-16T20:05:06.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:05:06.355 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T20:05:06.355 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zone1", 2021-09-16T20:05:06.356 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2021-09-16T20:05:06.356 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi109.front.sepia.ceph.com:8000" 2021-09-16T20:05:06.356 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T20:05:06.356 INFO:teuthology.orchestra.run.smithi118.stdout: "log_meta": "false", 2021-09-16T20:05:06.357 INFO:teuthology.orchestra.run.smithi118.stdout: "log_data": "true", 2021-09-16T20:05:06.357 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_index_max_shards": 11, 2021-09-16T20:05:06.357 INFO:teuthology.orchestra.run.smithi118.stdout: "read_only": "false", 2021-09-16T20:05:06.357 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_type": "", 2021-09-16T20:05:06.357 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from_all": "true", 2021-09-16T20:05:06.358 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from": [], 2021-09-16T20:05:06.358 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_zone": "" 2021-09-16T20:05:06.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:05:06.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:05:06.358 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "8df190b8-e36d-4ab9-bca1-30bba5bb5ec5", 2021-09-16T20:05:06.359 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zone3", 2021-09-16T20:05:06.359 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2021-09-16T20:05:06.359 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi109.front.sepia.ceph.com:8001" 2021-09-16T20:05:06.359 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T20:05:06.360 INFO:teuthology.orchestra.run.smithi118.stdout: "log_meta": "false", 2021-09-16T20:05:06.360 INFO:teuthology.orchestra.run.smithi118.stdout: "log_data": "true", 2021-09-16T20:05:06.360 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_index_max_shards": 11, 2021-09-16T20:05:06.360 INFO:teuthology.orchestra.run.smithi118.stdout: "read_only": "false", 2021-09-16T20:05:06.360 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_type": "", 2021-09-16T20:05:06.361 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from_all": "true", 2021-09-16T20:05:06.361 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from": [], 2021-09-16T20:05:06.361 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_zone": "" 2021-09-16T20:05:06.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:05:06.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:05:06.362 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "c6342467-0ed0-40c7-a5ab-491a56113d8e", 2021-09-16T20:05:06.362 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zone4", 2021-09-16T20:05:06.362 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2021-09-16T20:05:06.362 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2021-09-16T20:05:06.363 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T20:05:06.363 INFO:teuthology.orchestra.run.smithi118.stdout: "log_meta": "false", 2021-09-16T20:05:06.363 INFO:teuthology.orchestra.run.smithi118.stdout: "log_data": "true", 2021-09-16T20:05:06.363 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_index_max_shards": 11, 2021-09-16T20:05:06.363 INFO:teuthology.orchestra.run.smithi118.stdout: "read_only": "false", 2021-09-16T20:05:06.364 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_type": "pubsub", 2021-09-16T20:05:06.364 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from_all": "false", 2021-09-16T20:05:06.364 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from": [ 2021-09-16T20:05:06.364 INFO:teuthology.orchestra.run.smithi118.stdout: "test-zone1" 2021-09-16T20:05:06.364 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T20:05:06.365 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_zone": "" 2021-09-16T20:05:06.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:05:06.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:05:06.365 INFO:teuthology.orchestra.run.smithi118.stdout: "id": "ceba8a70-b881-4808-8acf-e8ddfa5f6daf", 2021-09-16T20:05:06.365 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test-zone2", 2021-09-16T20:05:06.366 INFO:teuthology.orchestra.run.smithi118.stdout: "endpoints": [ 2021-09-16T20:05:06.366 INFO:teuthology.orchestra.run.smithi118.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2021-09-16T20:05:06.366 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T20:05:06.366 INFO:teuthology.orchestra.run.smithi118.stdout: "log_meta": "false", 2021-09-16T20:05:06.366 INFO:teuthology.orchestra.run.smithi118.stdout: "log_data": "true", 2021-09-16T20:05:06.367 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_index_max_shards": 11, 2021-09-16T20:05:06.367 INFO:teuthology.orchestra.run.smithi118.stdout: "read_only": "false", 2021-09-16T20:05:06.367 INFO:teuthology.orchestra.run.smithi118.stdout: "tier_type": "", 2021-09-16T20:05:06.367 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from_all": "false", 2021-09-16T20:05:06.368 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_from": [], 2021-09-16T20:05:06.368 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_zone": "" 2021-09-16T20:05:06.368 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:05:06.368 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T20:05:06.368 INFO:teuthology.orchestra.run.smithi118.stdout: "placement_targets": [ 2021-09-16T20:05:06.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:05:06.369 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default-placement", 2021-09-16T20:05:06.369 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2021-09-16T20:05:06.369 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_classes": [ 2021-09-16T20:05:06.369 INFO:teuthology.orchestra.run.smithi118.stdout: "STANDARD" 2021-09-16T20:05:06.369 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T20:05:06.370 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:05:06.370 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T20:05:06.370 INFO:teuthology.orchestra.run.smithi118.stdout: "default_placement": "default-placement", 2021-09-16T20:05:06.370 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T20:05:06.371 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_policy": { 2021-09-16T20:05:06.371 INFO:teuthology.orchestra.run.smithi118.stdout: "groups": [] 2021-09-16T20:05:06.371 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:05:06.371 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:05:06.371 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2021-09-16T20:05:06.371 INFO:teuthology.orchestra.run.smithi118.stdout: "short_zone_ids": [ 2021-09-16T20:05:06.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:05:06.373 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T20:05:06.373 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 3703685179 2021-09-16T20:05:06.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:05:06.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:05:06.374 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "8df190b8-e36d-4ab9-bca1-30bba5bb5ec5", 2021-09-16T20:05:06.374 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 3204282247 2021-09-16T20:05:06.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:05:06.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:05:06.375 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "c6342467-0ed0-40c7-a5ab-491a56113d8e", 2021-09-16T20:05:06.375 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 1461382917 2021-09-16T20:05:06.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:05:06.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:05:06.375 INFO:teuthology.orchestra.run.smithi118.stdout: "key": "ceba8a70-b881-4808-8acf-e8ddfa5f6daf", 2021-09-16T20:05:06.376 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 3287816121 2021-09-16T20:05:06.376 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:05:06.376 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T20:05:06.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:05:06.376 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zonegroup": "test-zonegroup", 2021-09-16T20:05:06.377 INFO:teuthology.orchestra.run.smithi118.stdout: "master_zone": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T20:05:06.377 INFO:teuthology.orchestra.run.smithi118.stdout: "period_config": { 2021-09-16T20:05:06.377 INFO:teuthology.orchestra.run.smithi118.stdout: "bucket_quota": { 2021-09-16T20:05:06.377 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": false, 2021-09-16T20:05:06.378 INFO:teuthology.orchestra.run.smithi118.stdout: "check_on_raw": false, 2021-09-16T20:05:06.378 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size": -1, 2021-09-16T20:05:06.378 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size_kb": 0, 2021-09-16T20:05:06.378 INFO:teuthology.orchestra.run.smithi118.stdout: "max_objects": -1 2021-09-16T20:05:06.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:05:06.379 INFO:teuthology.orchestra.run.smithi118.stdout: "user_quota": { 2021-09-16T20:05:06.379 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": false, 2021-09-16T20:05:06.379 INFO:teuthology.orchestra.run.smithi118.stdout: "check_on_raw": false, 2021-09-16T20:05:06.379 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size": -1, 2021-09-16T20:05:06.379 INFO:teuthology.orchestra.run.smithi118.stdout: "max_size_kb": 0, 2021-09-16T20:05:06.380 INFO:teuthology.orchestra.run.smithi118.stdout: "max_objects": -1 2021-09-16T20:05:06.380 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:05:06.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:05:06.380 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T20:05:06.380 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_name": "test-realm", 2021-09-16T20:05:06.381 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:05:06.381 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T20:05:06.381 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2021-09-16T20:06:06.383 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-45 2021-09-16T20:06:06.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:06.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:06.472 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:06:06.791 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:06:06.791 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:06:06.965 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T20:06:06.986 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T20:06:06.986 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:06:06.986 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:06:06.987 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T20:06:06.987 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:06:06.987 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:06:06.988 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:06:06.988 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T20:06:06.988 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:06:06.988 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:06:06.988 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631822766.419765_313.1", 2021-09-16T20:06:06.989 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T20:06:06.419765Z" 2021-09-16T20:06:06.989 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:06:06.989 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:06:06.989 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:06:06.990 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:17.633408Z" 2021-09-16T20:06:06.990 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:06:06.990 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T20:06:06.991 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T20:06:06.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:06.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:06.993 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:06:07.041 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:06:07.041 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:06:07.097 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T20:06:07.097 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T20:06:07.097 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T20:06:07.098 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T20:06:07.098 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T20:06:07.098 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:06:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T20:06:07.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:07.100 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T20:06:07.100 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:07.100 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:07.100 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:06:07.100 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:07.101 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T20:06:07.101 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:07.101 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:06:07.101 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:07.102 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:07.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:07.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:07.102 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T20:06:07.102 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:07.103 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:07.103 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T20:06:07.103 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:07.103 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T20:06:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:07.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T20:06:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:07.105 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631822389.517479_289.1", 2021-09-16T20:06:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:07.106 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:59:49.517479Z", 2021-09-16T20:06:07.107 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:07.107 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:07.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:07.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T20:06:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:06:07.108 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:07.109 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:07.109 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:07.109 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:06:07.109 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:07.110 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:07.110 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:07.110 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T20:06:07.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:07.110 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T20:06:07.111 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T20:06:07.111 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T20:06:07.111 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:07.111 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T20:06:07.113 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631822389.517479_289.1 2021-09-16T20:06:12.113 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:12.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:12.115 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:06:12.164 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:06:12.165 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:06:12.228 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T20:06:12.228 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T20:06:12.228 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T20:06:12.229 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T20:06:12.229 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T20:06:12.229 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:06:12.229 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:12.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:12.230 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T20:06:12.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:12.230 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T20:06:12.231 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:12.231 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:12.231 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:06:12.232 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:12.232 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T20:06:12.232 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:12.232 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:06:12.233 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:12.233 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:12.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:12.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:12.234 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T20:06:12.234 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:12.234 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:12.234 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T20:06:12.234 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:12.235 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:12.235 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:12.235 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T20:06:12.235 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:12.236 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:12.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:12.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:12.236 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T20:06:12.236 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:12.237 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:12.237 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631822389.517479_289.1", 2021-09-16T20:06:12.237 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:12.237 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:12.238 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:12.238 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:59:49.517479Z", 2021-09-16T20:06:12.238 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:12.238 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:12.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:12.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:12.239 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T20:06:12.239 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:12.239 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:12.240 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:06:12.240 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:12.240 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:12.240 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:12.240 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:06:12.241 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:12.241 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:12.241 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:12.241 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T20:06:12.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:12.242 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T20:06:12.242 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T20:06:12.242 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T20:06:12.243 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:12.243 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T20:06:12.244 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631822389.517479_289.1 2021-09-16T20:06:17.246 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:17.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:17.247 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:06:17.295 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:06:17.296 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:06:17.351 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T20:06:17.351 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T20:06:17.351 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T20:06:17.352 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T20:06:17.352 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T20:06:17.352 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:06:17.352 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:17.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:17.353 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T20:06:17.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:17.353 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T20:06:17.353 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:17.354 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:17.354 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:06:17.354 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:17.354 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T20:06:17.354 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:17.355 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:06:17.355 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:17.355 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:17.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:17.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:17.356 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T20:06:17.356 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:17.356 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:17.357 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T20:06:17.357 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:17.357 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:17.357 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:17.357 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T20:06:17.358 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:17.358 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:17.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:17.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:17.359 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T20:06:17.359 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:17.359 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:17.359 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631822389.517479_289.1", 2021-09-16T20:06:17.360 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:17.360 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:17.360 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:17.360 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:59:49.517479Z", 2021-09-16T20:06:17.361 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:17.361 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:17.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:17.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:17.362 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T20:06:17.362 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:17.362 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:17.362 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:06:17.363 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:17.363 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:17.363 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:17.364 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:06:17.364 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:17.364 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:17.364 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:17.365 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T20:06:17.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:17.365 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T20:06:17.365 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T20:06:17.365 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T20:06:17.366 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:17.366 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T20:06:17.367 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631822389.517479_289.1 2021-09-16T20:06:22.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:22.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:22.370 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:06:22.417 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:06:22.417 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:06:22.472 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T20:06:22.472 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T20:06:22.473 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T20:06:22.473 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T20:06:22.473 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T20:06:22.473 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:06:22.473 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:22.474 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:22.474 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T20:06:22.474 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:22.474 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T20:06:22.475 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:22.475 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:22.475 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:06:22.475 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:22.476 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T20:06:22.476 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:22.476 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:06:22.476 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:22.476 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T20:06:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:22.478 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T20:06:22.478 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:22.478 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:22.478 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:22.479 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T20:06:22.479 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:22.479 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:22.479 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:22.479 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:22.480 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T20:06:22.480 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:22.480 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:22.480 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631822389.517479_289.1", 2021-09-16T20:06:22.480 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:22.481 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:22.481 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:22.481 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:59:49.517479Z", 2021-09-16T20:06:22.481 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:22.482 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:22.482 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:22.482 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:22.482 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T20:06:22.482 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:22.483 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:22.483 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:06:22.483 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:22.483 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:22.483 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:22.484 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:06:22.484 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:22.484 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:22.484 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:22.484 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T20:06:22.485 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:22.485 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T20:06:22.485 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T20:06:22.485 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T20:06:22.486 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:22.486 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T20:06:22.487 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631822389.517479_289.1 2021-09-16T20:06:27.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:27.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:27.489 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:06:27.537 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:06:27.538 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:06:27.593 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T20:06:27.593 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T20:06:27.593 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T20:06:27.594 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T20:06:27.594 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T20:06:27.594 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:06:27.594 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:27.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:27.595 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T20:06:27.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:27.595 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T20:06:27.596 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:27.596 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:27.596 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:06:27.596 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:27.596 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T20:06:27.597 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:27.597 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:06:27.597 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:27.597 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:27.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:27.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:27.598 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T20:06:27.598 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:27.598 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:27.599 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T20:06:27.599 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:27.599 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:27.599 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:27.600 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T20:06:27.600 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:27.600 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:27.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:27.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:27.601 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T20:06:27.601 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:27.601 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:27.601 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631822389.517479_289.1", 2021-09-16T20:06:27.602 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:27.602 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:27.602 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:27.602 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:59:49.517479Z", 2021-09-16T20:06:27.602 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:27.603 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:27.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:27.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:27.603 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T20:06:27.604 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:27.604 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:27.604 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:06:27.604 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:27.604 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:27.605 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:27.605 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:06:27.605 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:27.605 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:27.605 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:27.606 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T20:06:27.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:27.606 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T20:06:27.606 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T20:06:27.607 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T20:06:27.607 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:27.607 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T20:06:27.608 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631822389.517479_289.1 2021-09-16T20:06:32.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:32.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:32.611 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:06:32.658 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:06:32.659 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:06:32.717 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T20:06:32.717 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T20:06:32.718 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T20:06:32.718 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T20:06:32.718 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T20:06:32.719 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:06:32.719 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:32.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:32.719 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T20:06:32.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:32.720 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T20:06:32.720 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:32.720 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:32.720 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:06:32.721 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:32.721 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T20:06:32.721 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:32.721 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:06:32.722 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:32.722 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:32.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:32.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:32.723 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T20:06:32.723 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:32.723 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:32.723 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T20:06:32.723 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:32.724 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:32.724 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:32.724 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T20:06:32.724 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:32.724 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:32.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:32.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:32.725 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T20:06:32.725 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:32.726 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:32.726 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631822389.517479_289.1", 2021-09-16T20:06:32.726 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:32.726 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:32.727 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:32.727 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:59:49.517479Z", 2021-09-16T20:06:32.727 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:32.727 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:32.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:32.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:32.728 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T20:06:32.728 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:32.728 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:32.729 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:06:32.729 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:32.729 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:32.729 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:32.730 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:06:32.730 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:32.730 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:32.730 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:32.730 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T20:06:32.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:32.731 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T20:06:32.731 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T20:06:32.731 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T20:06:32.732 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:32.732 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T20:06:32.733 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631822389.517479_289.1 2021-09-16T20:06:37.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:37.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:37.736 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:06:37.784 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:06:37.785 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:06:37.840 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T20:06:37.840 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T20:06:37.840 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T20:06:37.841 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T20:06:37.841 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T20:06:37.841 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:06:37.841 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:37.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:37.842 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T20:06:37.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:37.842 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T20:06:37.842 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:37.843 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:37.843 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:06:37.843 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:37.843 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T20:06:37.844 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:37.844 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:06:37.844 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:37.844 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:37.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:37.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:37.845 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T20:06:37.845 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:37.845 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:37.846 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T20:06:37.846 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:37.846 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:37.846 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:37.846 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T20:06:37.847 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:37.847 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:37.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:37.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:37.848 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T20:06:37.848 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:37.848 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:37.848 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631822389.517479_289.1", 2021-09-16T20:06:37.848 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:37.849 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:37.849 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:37.849 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:59:49.517479Z", 2021-09-16T20:06:37.850 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:37.850 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:37.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:37.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:37.850 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T20:06:37.851 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:37.851 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:37.851 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:06:37.851 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:37.852 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:37.852 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:37.852 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:06:37.852 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:37.852 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:37.853 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:37.853 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T20:06:37.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:37.853 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T20:06:37.853 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T20:06:37.854 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T20:06:37.854 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:37.854 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T20:06:37.855 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631822389.517479_289.1 2021-09-16T20:06:42.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:42.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:42.859 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:06:42.907 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:06:42.907 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:06:42.962 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T20:06:42.962 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T20:06:42.962 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T20:06:42.963 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T20:06:42.963 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T20:06:42.963 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:06:42.963 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T20:06:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T20:06:42.964 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:42.965 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:42.965 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:06:42.965 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:42.965 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T20:06:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:06:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:42.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T20:06:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:42.967 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T20:06:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T20:06:42.968 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:42.969 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T20:06:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631822389.517479_289.1", 2021-09-16T20:06:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:42.970 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:42.971 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:42.971 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:59:49.517479Z", 2021-09-16T20:06:42.971 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:42.971 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T20:06:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:42.972 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:42.973 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:06:42.973 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:42.973 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:42.973 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:42.973 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:06:42.974 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:42.974 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:42.974 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:42.974 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T20:06:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T20:06:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T20:06:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T20:06:42.975 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:42.976 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T20:06:42.977 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631822389.517479_289.1 2021-09-16T20:06:47.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:47.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:47.979 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:06:48.026 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:06:48.027 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:06:48.082 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T20:06:48.082 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T20:06:48.082 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T20:06:48.083 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T20:06:48.083 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T20:06:48.083 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:06:48.083 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:48.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:48.084 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T20:06:48.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:48.084 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T20:06:48.085 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:48.085 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:48.085 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:06:48.085 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:48.085 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T20:06:48.086 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:48.086 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:06:48.086 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:48.086 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:48.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:48.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:48.087 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T20:06:48.087 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:48.087 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:48.088 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T20:06:48.088 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:48.088 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:48.088 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:48.088 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T20:06:48.088 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:48.089 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:48.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:48.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:48.089 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T20:06:48.089 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:48.090 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:48.090 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631822389.517479_289.1", 2021-09-16T20:06:48.090 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:48.090 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:48.090 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:48.091 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:59:49.517479Z", 2021-09-16T20:06:48.091 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:48.091 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:48.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:48.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:48.092 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T20:06:48.092 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:48.092 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:48.092 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:06:48.093 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:48.093 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:48.093 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:48.093 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:06:48.094 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:48.094 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:48.094 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:48.094 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T20:06:48.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:48.095 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T20:06:48.095 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T20:06:48.095 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T20:06:48.095 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:48.096 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T20:06:48.097 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631822389.517479_289.1 2021-09-16T20:06:53.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:53.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:53.099 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:06:53.148 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:06:53.148 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:06:53.211 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T20:06:53.212 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T20:06:53.212 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T20:06:53.212 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T20:06:53.212 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T20:06:53.213 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:06:53.213 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:53.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:53.213 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T20:06:53.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:53.214 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T20:06:53.214 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:53.214 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:53.214 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:06:53.215 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:53.215 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T20:06:53.215 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:53.215 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:06:53.215 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:53.216 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:53.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:53.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:53.216 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T20:06:53.216 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:53.217 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:53.217 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T20:06:53.217 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:53.217 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:53.217 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:53.218 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T20:06:53.218 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:53.218 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:53.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:53.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:53.219 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T20:06:53.219 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:53.219 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:53.219 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631822389.517479_289.1", 2021-09-16T20:06:53.219 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:53.220 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:53.220 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:53.220 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:59:49.517479Z", 2021-09-16T20:06:53.220 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:53.220 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:53.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:53.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:53.222 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T20:06:53.222 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:53.222 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:53.222 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:06:53.222 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:53.223 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:53.223 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:53.223 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:06:53.223 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:53.223 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:53.224 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:53.224 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T20:06:53.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:53.224 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T20:06:53.224 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T20:06:53.225 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T20:06:53.225 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:53.225 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T20:06:53.226 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631822389.517479_289.1 2021-09-16T20:06:58.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:58.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:06:58.228 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:06:58.276 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:06:58.276 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:06:58.331 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T20:06:58.331 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T20:06:58.331 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T20:06:58.331 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T20:06:58.332 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T20:06:58.332 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:06:58.332 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:58.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:58.333 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T20:06:58.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:58.333 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T20:06:58.333 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:58.334 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:58.334 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:06:58.334 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:58.334 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T20:06:58.334 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:58.335 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:06:58.335 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:58.335 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:58.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:58.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:58.336 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T20:06:58.336 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:58.336 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:58.336 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T20:06:58.337 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:58.337 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:58.337 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:58.337 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T20:06:58.337 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:58.338 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:58.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:58.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:58.338 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T20:06:58.339 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:58.339 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:58.339 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631822389.517479_289.1", 2021-09-16T20:06:58.339 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:58.339 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:58.340 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:58.340 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:59:49.517479Z", 2021-09-16T20:06:58.340 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:58.340 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:58.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:58.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:06:58.341 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T20:06:58.341 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:06:58.341 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:06:58.342 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:06:58.342 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:06:58.342 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:06:58.342 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:06:58.343 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:06:58.343 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:06:58.343 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:58.343 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:58.343 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T20:06:58.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:06:58.344 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T20:06:58.344 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T20:06:58.344 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T20:06:58.345 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:06:58.345 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T20:06:58.346 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631822389.517479_289.1 2021-09-16T20:07:03.347 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:03.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:03.348 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:07:03.397 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:07:03.398 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:07:03.455 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T20:07:03.455 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T20:07:03.455 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T20:07:03.455 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T20:07:03.456 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T20:07:03.456 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:07:03.456 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:07:03.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:07:03.457 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T20:07:03.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:07:03.457 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T20:07:03.457 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:07:03.457 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:07:03.458 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:07:03.458 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:07:03.458 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T20:07:03.458 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:07:03.459 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:07:03.459 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:07:03.459 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:07:03.459 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:07:03.460 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:07:03.460 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T20:07:03.460 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:07:03.460 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:07:03.460 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T20:07:03.461 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:07:03.461 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:07:03.461 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:07:03.461 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T20:07:03.462 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:07:03.462 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:07:03.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:07:03.462 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:07:03.463 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T20:07:03.463 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:07:03.463 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:07:03.463 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631822389.517479_289.1", 2021-09-16T20:07:03.463 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:07:03.464 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:07:03.464 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:07:03.464 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:59:49.517479Z", 2021-09-16T20:07:03.464 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:07:03.465 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:07:03.465 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:07:03.465 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:07:03.465 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T20:07:03.465 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:07:03.466 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:07:03.466 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:07:03.466 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:07:03.466 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:07:03.466 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:07:03.467 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:07:03.467 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:07:03.467 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:07:03.467 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:07:03.468 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T20:07:03.468 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:07:03.468 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T20:07:03.468 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T20:07:03.468 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T20:07:03.469 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:07:03.469 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T20:07:03.470 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631822389.517479_289.1 2021-09-16T20:07:06.527 INFO:tasks.rgw.c2.client.1.smithi118.stdout:2021-09-16T20:07:06.527+0000 7f2fdeb7f700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-16T20:07:08.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:08.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:08.473 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:07:08.519 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:07:08.519 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:07:08.573 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T20:07:08.574 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T20:07:08.574 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T20:07:08.574 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T20:07:08.575 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T20:07:08.575 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:07:08.575 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:07:08.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:07:08.575 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T20:07:08.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:07:08.576 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T20:07:08.576 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:07:08.576 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:07:08.576 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:07:08.577 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:07:08.577 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T20:07:08.577 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:07:08.577 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:07:08.577 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:07:08.578 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:07:08.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:07:08.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:07:08.578 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T20:07:08.579 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:07:08.579 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:07:08.579 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T20:07:08.579 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:07:08.579 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:07:08.580 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:07:08.580 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T20:07:08.580 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:07:08.580 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:07:08.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:07:08.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:07:08.581 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T20:07:08.581 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:07:08.581 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:07:08.581 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631822766.419765_313.1", 2021-09-16T20:07:08.582 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:07:08.582 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:07:08.582 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:07:08.582 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T20:06:06.419765Z", 2021-09-16T20:07:08.583 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:07:08.583 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:07:08.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:07:08.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:07:08.583 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T20:07:08.584 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:07:08.584 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:07:08.584 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:07:08.584 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:07:08.584 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:07:08.585 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:07:08.585 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:07:08.585 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:07:08.585 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:07:08.585 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:07:08.586 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T20:07:08.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:07:08.586 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T20:07:08.586 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T20:07:08.587 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T20:07:08.587 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:07:08.587 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T20:07:08.588 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T20:07:08.589 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T20:07:08.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:08.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:08.590 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:07:08.633 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:07:08.633 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:07:08.692 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:07:08.692 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:07:08.693 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:07:08.693 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:07:08.693 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:07:08.693 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:07:08.694 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:08.694 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:08.694 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:07:08.694 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:08.695 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:07:08.695 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:08.695 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:08.695 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:07:08.695 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:08.696 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:07:08.696 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:08.696 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:07:08.697 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:08.697 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:08.697 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:08.697 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:08.697 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:07:08.698 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:08.698 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:08.698 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:07:08.698 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:08.699 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:08.699 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:08.699 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:07:08.699 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:08.700 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:08.700 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:08.700 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:08.700 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:07:08.700 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:08.701 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:08.701 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.791325_211.1", 2021-09-16T20:07:08.701 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:08.701 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:08.701 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:08.702 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.791325Z", 2021-09-16T20:07:08.702 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:08.702 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:08.702 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:08.703 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:08.703 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:07:08.703 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:08.703 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:08.704 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:07:08.704 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:08.704 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:08.704 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:08.704 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:07:08.705 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:08.705 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:08.705 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:08.705 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:07:08.705 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:08.706 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:07:08.706 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:07:08.706 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:07:08.707 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:08.707 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:07:08.708 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.791325_211.1 2021-09-16T20:07:13.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:13.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:13.712 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:07:13.762 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:07:13.762 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:07:13.829 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:07:13.830 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:07:13.830 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:07:13.830 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:07:13.830 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:07:13.831 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:07:13.831 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:13.831 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:13.831 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:07:13.831 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:13.832 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:07:13.832 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:13.832 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:13.832 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:07:13.833 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:13.833 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:07:13.833 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:13.834 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:07:13.834 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:13.834 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:13.834 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:13.835 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:13.835 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:07:13.835 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:13.835 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:13.836 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:07:13.836 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:13.836 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:13.836 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:13.836 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:07:13.837 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:13.837 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:13.837 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:13.837 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:13.837 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:07:13.838 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:13.838 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:13.838 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.765733_208.1", 2021-09-16T20:07:13.838 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:13.839 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:13.839 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:13.839 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.765733Z", 2021-09-16T20:07:13.839 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:13.839 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:13.840 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:13.840 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:13.840 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:07:13.840 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:13.840 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:13.841 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:07:13.841 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:13.841 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:13.841 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:13.842 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:07:13.842 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:13.842 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:13.842 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:13.843 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:07:13.843 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:13.843 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:07:13.843 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:07:13.844 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:07:13.844 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:13.844 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:07:13.845 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.765733_208.1 2021-09-16T20:07:18.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:18.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:18.848 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:07:18.924 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:07:18.925 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:07:19.217 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:07:19.217 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:07:19.217 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:07:19.217 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:07:19.218 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:07:19.218 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:07:19.218 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:19.218 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:19.219 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:07:19.219 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:19.219 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:07:19.219 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:19.219 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:19.220 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:07:19.220 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:19.220 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:07:19.220 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:19.221 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:07:19.221 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:19.221 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:19.221 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:19.221 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:19.222 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:07:19.222 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:19.222 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:19.222 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:07:19.222 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:19.223 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:19.223 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:19.223 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:07:19.223 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:19.224 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:19.224 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:19.224 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:19.224 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:07:19.224 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:19.225 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:19.225 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.768089_209.1", 2021-09-16T20:07:19.225 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:19.225 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:19.225 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:19.226 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.768089Z", 2021-09-16T20:07:19.226 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:19.226 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:19.226 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:19.227 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:19.227 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:07:19.227 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:19.227 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:19.227 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:07:19.228 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:19.228 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:19.228 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:19.228 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:07:19.228 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:19.229 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:19.229 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:19.229 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:07:19.229 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:19.229 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:07:19.230 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:07:19.230 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:07:19.230 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:19.230 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:07:19.232 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.768089_209.1 2021-09-16T20:07:24.233 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:24.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:24.234 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:07:24.302 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:07:24.303 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:07:24.389 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:07:24.389 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:07:24.390 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:07:24.390 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:07:24.390 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:07:24.390 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:07:24.391 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:24.391 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:24.391 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:07:24.391 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:24.392 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:07:24.392 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:24.392 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:24.392 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:07:24.392 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:24.393 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:07:24.393 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:24.393 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:07:24.393 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:24.394 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:24.394 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:24.394 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:24.394 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:07:24.395 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:24.395 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:24.395 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:07:24.395 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:24.395 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:24.396 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:24.396 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:07:24.396 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:24.396 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:24.397 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:24.397 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:24.397 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:07:24.397 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:24.398 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:24.398 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.839033_212.1", 2021-09-16T20:07:24.398 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:24.398 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:24.398 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:24.399 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.839033Z", 2021-09-16T20:07:24.399 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:24.399 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:24.399 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:24.400 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:24.400 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:07:24.400 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:24.400 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:24.400 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:07:24.401 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:24.401 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:24.401 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:24.401 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:07:24.402 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:24.402 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:24.402 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:24.402 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:07:24.403 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:24.403 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:07:24.403 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:07:24.403 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:07:24.403 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:24.404 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:07:24.405 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.839033_212.1 2021-09-16T20:07:29.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:29.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:29.408 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:07:29.451 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:07:29.451 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:07:29.510 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:07:29.511 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:07:29.511 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:07:29.511 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:07:29.511 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:07:29.512 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:07:29.512 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:29.512 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:29.512 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:07:29.512 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:29.513 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:07:29.513 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:29.513 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:29.513 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:07:29.514 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:29.514 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:07:29.514 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:29.514 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:07:29.514 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:29.515 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:29.515 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:29.515 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:29.515 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:07:29.516 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:29.516 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:29.516 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:07:29.516 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:29.516 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:29.517 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:29.517 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:07:29.517 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:29.517 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:29.518 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:29.518 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:29.518 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:07:29.518 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:29.519 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:29.519 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821519.160039_228.1", 2021-09-16T20:07:29.519 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:29.519 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:29.520 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:29.520 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:19.160039Z", 2021-09-16T20:07:29.520 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:29.520 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:29.520 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:29.521 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:29.521 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:07:29.521 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:29.521 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:29.522 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:07:29.522 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:29.522 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:29.522 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:29.522 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:07:29.523 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:29.523 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:29.523 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:29.523 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:07:29.524 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:29.524 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:07:29.524 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:07:29.524 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:07:29.524 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:29.525 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:07:29.526 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821519.160039_228.1 2021-09-16T20:07:34.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:34.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:34.529 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:07:34.683 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:07:34.683 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:07:34.746 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:07:34.746 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:07:34.747 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:07:34.747 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:07:34.747 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:07:34.747 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:07:34.748 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:34.748 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:34.748 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:07:34.748 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:34.748 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:07:34.749 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:34.749 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:34.749 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:07:34.749 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:34.750 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:07:34.750 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:34.750 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:07:34.750 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:34.750 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:34.751 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:34.751 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:34.751 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:07:34.751 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:34.752 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:34.752 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:07:34.752 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:34.752 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:34.753 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:34.753 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:07:34.753 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:34.753 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:34.753 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:34.754 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:34.754 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:07:34.754 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:34.754 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:34.755 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.891592_214.1", 2021-09-16T20:07:34.755 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:34.755 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:34.755 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:34.755 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.891592Z", 2021-09-16T20:07:34.756 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:34.756 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:34.756 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:34.756 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:34.757 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:07:34.757 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:34.757 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:34.757 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:07:34.757 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:34.758 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:34.758 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:34.758 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:07:34.758 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:34.759 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:34.759 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:34.759 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:07:34.759 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:34.759 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:07:34.760 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:07:34.760 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:07:34.760 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:34.760 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:07:34.762 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.891592_214.1 2021-09-16T20:07:39.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:39.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:39.764 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:07:39.834 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:07:39.835 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:07:40.240 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:07:40.240 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:07:40.241 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:07:40.241 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:07:40.241 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:07:40.241 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:07:40.242 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:40.242 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:40.242 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:07:40.242 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:40.243 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:07:40.243 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:40.243 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:40.243 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:07:40.243 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:40.244 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:07:40.244 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:40.244 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:07:40.245 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:40.245 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:40.245 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:40.245 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:40.246 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:07:40.246 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:40.246 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:40.247 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:07:40.247 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:40.247 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:40.247 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:40.248 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:07:40.248 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:40.248 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:40.248 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:40.248 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:40.249 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:07:40.249 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:40.249 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:40.249 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.788603_210.1", 2021-09-16T20:07:40.249 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:40.250 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:40.250 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:40.250 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.788603Z", 2021-09-16T20:07:40.250 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:40.251 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:40.251 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:40.251 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:40.251 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:07:40.252 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:40.252 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:40.252 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:07:40.252 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:40.252 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:40.253 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:40.253 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:07:40.253 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:40.253 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:40.254 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:40.254 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:07:40.254 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:40.254 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:07:40.254 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:07:40.255 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:07:40.255 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:40.255 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:07:40.256 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.788603_210.1 2021-09-16T20:07:45.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:45.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:45.259 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:07:45.319 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:07:45.319 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:07:45.409 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:07:45.410 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:07:45.410 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:07:45.410 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:07:45.410 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:07:45.411 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:07:45.411 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:45.411 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:45.411 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:07:45.411 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:45.412 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:07:45.412 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:45.412 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:45.412 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:07:45.413 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:45.413 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:07:45.413 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:45.413 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:07:45.413 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:45.414 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:45.414 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:45.414 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:45.414 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:07:45.414 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:45.415 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:45.415 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:07:45.415 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:45.415 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:45.416 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:45.416 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:07:45.416 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:45.416 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:45.416 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:45.417 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:45.417 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:07:45.417 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:45.417 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:45.417 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821520.221740_230.1", 2021-09-16T20:07:45.418 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:45.418 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:45.418 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:45.418 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:20.221740Z", 2021-09-16T20:07:45.419 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:45.419 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:45.419 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:45.419 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:45.419 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:07:45.420 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:45.420 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:45.420 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:07:45.420 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:45.421 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:45.421 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:45.421 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:07:45.421 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:45.422 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:45.422 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:45.422 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:07:45.422 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:45.423 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:07:45.423 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:07:45.423 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:07:45.423 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:45.423 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:07:45.425 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821520.221740_230.1 2021-09-16T20:07:50.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:50.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:50.427 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:07:50.472 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:07:50.472 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:07:50.523 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:07:50.524 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:07:50.524 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:07:50.524 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:07:50.524 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:07:50.524 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:07:50.525 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:50.525 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:50.525 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:07:50.525 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:50.525 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:07:50.526 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:50.526 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:50.526 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:07:50.526 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:50.527 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:07:50.527 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:50.527 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:07:50.527 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:50.527 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:50.528 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:50.528 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:50.528 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:07:50.528 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:50.528 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:50.529 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:07:50.529 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:50.529 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:50.529 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:50.529 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:07:50.530 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:50.530 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:50.530 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:50.530 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:50.530 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:07:50.531 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:50.531 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:50.531 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.791325_211.1", 2021-09-16T20:07:50.531 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:50.531 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:50.532 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:50.532 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.791325Z", 2021-09-16T20:07:50.532 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:50.532 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:50.532 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:50.533 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:50.533 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:07:50.533 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:50.533 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:50.533 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:07:50.534 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:50.534 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:50.534 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:50.534 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:07:50.534 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:50.535 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:50.535 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:50.535 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:07:50.535 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:50.535 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:07:50.536 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:07:50.536 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:07:50.536 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:50.536 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:07:50.537 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.791325_211.1 2021-09-16T20:07:55.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:55.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:07:55.540 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:07:55.757 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:07:55.758 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:07:55.829 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:07:55.829 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:07:55.830 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:07:55.830 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:07:55.830 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:07:55.830 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:07:55.831 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:55.831 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:55.831 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:07:55.831 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:55.832 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:07:55.832 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:55.832 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:55.832 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:07:55.832 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:55.833 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:07:55.833 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:55.833 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:07:55.833 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:55.834 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:55.834 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:55.834 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:55.834 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:07:55.835 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:55.835 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:55.835 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:07:55.835 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:55.835 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:55.836 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:55.836 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:07:55.836 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:55.836 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:55.837 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:55.837 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:55.837 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:07:55.837 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:55.838 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:55.838 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.765733_208.1", 2021-09-16T20:07:55.838 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:55.838 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:55.838 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:55.839 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.765733Z", 2021-09-16T20:07:55.839 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:55.839 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:55.839 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:55.840 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:07:55.840 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:07:55.840 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:07:55.840 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:07:55.841 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:07:55.841 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:07:55.841 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:07:55.841 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:07:55.841 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:07:55.842 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:07:55.842 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:55.842 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:55.842 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:07:55.843 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:07:55.843 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:07:55.843 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:07:55.843 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:07:55.844 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:07:55.844 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:07:55.845 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.765733_208.1 2021-09-16T20:08:00.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:00.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:00.848 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:08:00.930 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:08:00.930 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:08:01.053 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:08:01.053 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:08:01.053 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:08:01.054 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:08:01.054 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:08:01.054 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:08:01.054 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:01.055 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:01.055 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:08:01.055 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:01.055 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:08:01.056 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:01.056 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:01.056 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:08:01.056 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:01.056 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:08:01.057 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:01.057 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:08:01.057 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:01.058 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:01.058 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:01.058 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:01.058 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:08:01.059 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:01.059 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:01.059 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:08:01.059 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:01.059 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:01.060 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:01.060 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:08:01.060 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:01.060 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:01.060 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:01.061 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:01.061 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:08:01.061 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:01.061 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:01.062 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.765733_208.1", 2021-09-16T20:08:01.062 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:01.062 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:01.062 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:01.062 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.765733Z", 2021-09-16T20:08:01.063 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:01.063 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:01.063 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:01.064 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:01.064 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:08:01.064 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:01.064 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:01.064 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:08:01.065 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:01.065 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:01.065 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:01.065 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:08:01.066 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:01.066 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:01.066 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:01.066 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:08:01.067 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:01.067 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:08:01.067 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:08:01.067 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:08:01.068 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:01.068 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:08:01.069 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.765733_208.1 2021-09-16T20:08:06.071 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:06.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:06.072 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:08:06.145 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:08:06.145 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:08:06.263 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:08:06.263 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:08:06.264 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:08:06.264 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:08:06.264 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:08:06.264 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:08:06.264 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:06.265 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:06.265 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:08:06.265 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:06.265 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:08:06.266 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:06.266 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:06.266 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:08:06.266 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:06.267 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:08:06.267 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:06.267 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:08:06.267 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:06.268 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:06.268 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:06.268 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:06.268 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:08:06.269 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:06.269 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:06.269 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:08:06.269 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:06.269 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:06.270 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:06.270 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:08:06.270 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:06.270 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:06.271 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:06.271 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:06.271 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:08:06.271 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:06.271 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:06.272 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.765733_208.1", 2021-09-16T20:08:06.272 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:06.272 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:06.272 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:06.273 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.765733Z", 2021-09-16T20:08:06.273 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:06.273 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:06.273 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:06.274 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:06.274 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:08:06.274 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:06.274 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:06.274 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:08:06.275 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:06.275 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:06.275 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:06.275 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:08:06.276 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:06.276 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:06.276 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:06.276 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:08:06.276 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:06.277 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:08:06.277 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:08:06.277 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:08:06.277 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:06.278 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:08:06.279 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.765733_208.1 2021-09-16T20:08:11.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:11.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:11.282 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:08:11.325 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:08:11.326 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:08:11.377 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:08:11.377 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:08:11.377 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:08:11.378 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:08:11.378 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:08:11.378 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:08:11.378 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:11.378 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:11.379 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:08:11.379 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:11.379 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:08:11.379 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:11.380 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:11.380 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:08:11.380 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:11.380 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:08:11.381 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:11.381 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:08:11.381 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:11.381 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:11.381 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:11.382 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:11.382 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:08:11.382 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:11.382 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:11.383 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:08:11.383 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:11.383 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:11.383 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:11.383 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:08:11.384 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:11.384 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:11.384 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:11.384 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:11.385 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:08:11.385 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:11.385 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:11.385 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.957515_216.1", 2021-09-16T20:08:11.385 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:11.386 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:11.386 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:11.386 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.957515Z", 2021-09-16T20:08:11.386 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:11.387 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:11.387 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:11.387 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:11.387 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:08:11.387 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:11.388 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:11.388 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:08:11.388 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:11.388 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:11.389 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:11.389 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:08:11.389 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:11.389 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:11.389 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:11.390 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:08:11.390 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:11.390 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:08:11.390 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:08:11.391 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:08:11.391 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:11.391 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:08:11.393 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.957515_216.1 2021-09-16T20:08:16.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:16.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:16.396 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:08:16.441 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:08:16.442 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:08:16.506 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:08:16.506 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:08:16.507 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:08:16.507 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:08:16.507 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:08:16.508 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:08:16.508 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:16.508 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:16.508 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:08:16.509 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:16.509 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:08:16.509 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:16.509 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:16.510 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:08:16.510 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:16.510 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:08:16.510 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:16.510 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:08:16.511 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:16.511 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:16.511 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:16.511 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:16.512 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:08:16.512 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:16.512 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:16.512 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:08:16.513 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:16.513 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:16.513 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:16.513 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:08:16.513 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:16.514 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:16.514 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:16.514 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:16.514 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:08:16.515 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:16.515 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:16.515 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.765733_208.1", 2021-09-16T20:08:16.515 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:16.516 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:16.516 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:16.516 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.765733Z", 2021-09-16T20:08:16.516 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:16.516 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:16.517 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:16.517 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:16.517 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:08:16.517 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:16.517 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:16.518 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:08:16.518 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:16.518 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:16.518 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:16.518 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:08:16.519 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:16.519 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:16.519 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:16.519 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:08:16.522 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:16.522 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:08:16.522 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:08:16.522 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:08:16.523 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:16.523 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:08:16.524 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.765733_208.1 2021-09-16T20:08:21.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:21.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:21.527 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:08:21.610 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:08:21.610 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:08:21.924 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:08:21.925 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:08:21.925 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:08:21.925 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:08:21.926 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:08:21.926 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:08:21.926 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:21.926 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:21.927 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:08:21.927 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:21.927 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:08:21.927 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:21.928 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:21.928 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:08:21.928 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:21.929 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:08:21.929 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:21.929 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:08:21.929 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:21.930 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:21.930 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:21.930 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:21.930 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:08:21.930 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:21.931 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:21.931 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:08:21.931 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:21.931 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:21.931 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:21.932 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:08:21.932 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:21.932 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:21.932 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:21.933 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:21.933 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:08:21.933 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:21.933 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:21.933 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.765733_208.1", 2021-09-16T20:08:21.934 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:21.934 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:21.934 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:21.934 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.765733Z", 2021-09-16T20:08:21.935 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:21.935 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:21.935 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:21.935 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:21.935 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:08:21.936 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:21.936 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:21.936 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:08:21.936 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:21.937 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:21.937 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:21.937 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:08:21.937 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:21.937 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:21.938 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:21.938 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:08:21.938 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:21.938 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:08:21.938 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:08:21.939 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:08:21.939 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:21.939 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:08:21.940 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.765733_208.1 2021-09-16T20:08:26.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:26.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:26.943 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:08:27.012 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:08:27.012 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:08:27.140 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:08:27.141 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:08:27.141 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:08:27.141 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:08:27.142 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:08:27.142 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:08:27.142 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:27.142 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:27.143 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:08:27.143 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:27.143 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:08:27.143 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:27.144 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:27.144 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:08:27.144 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:27.144 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:08:27.144 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:27.145 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:08:27.145 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:27.145 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:27.145 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:27.146 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:27.146 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:08:27.146 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:27.146 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:27.146 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:08:27.147 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:27.147 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:27.147 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:27.147 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:08:27.147 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:27.148 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:27.148 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:27.148 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:27.148 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:08:27.149 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:27.149 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:27.149 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.768089_209.1", 2021-09-16T20:08:27.149 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:27.149 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:27.150 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:27.150 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.768089Z", 2021-09-16T20:08:27.150 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:27.150 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:27.150 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:27.151 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:27.151 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:08:27.151 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:27.151 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:27.151 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:08:27.152 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:27.152 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:27.152 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:27.152 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:08:27.152 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:27.153 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:27.153 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:27.153 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:08:27.153 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:27.154 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:08:27.154 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:08:27.154 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:08:27.154 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:27.154 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:08:27.156 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.768089_209.1 2021-09-16T20:08:32.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:32.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:32.158 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:08:32.202 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:08:32.203 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:08:32.259 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:08:32.259 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:08:32.259 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:08:32.259 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:08:32.260 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:08:32.260 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:08:32.260 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:32.260 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:32.260 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:08:32.261 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:32.261 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:08:32.261 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:32.261 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:32.261 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:08:32.262 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:32.262 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:08:32.262 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:32.262 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:08:32.263 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:32.263 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:32.263 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:32.263 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:32.263 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:08:32.264 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:32.264 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:32.264 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:08:32.264 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:32.264 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:32.265 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:32.265 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:08:32.265 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:32.265 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:32.265 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:32.266 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:32.266 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:08:32.266 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:32.266 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:32.267 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:08:32.267 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:32.267 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:32.267 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:32.267 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:08:32.268 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:32.268 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:32.268 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:32.268 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:32.269 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:08:32.269 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:32.269 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:32.269 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:08:32.269 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:32.270 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:32.270 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:32.270 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:08:32.270 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:32.270 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:32.271 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:32.271 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:08:32.271 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:32.271 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:08:32.271 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:08:32.272 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:08:32.272 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:32.272 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:08:32.273 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:08:37.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:37.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:37.276 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:08:37.323 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:08:37.324 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:08:37.388 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:08:37.388 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:08:37.388 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:08:37.388 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:08:37.389 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:08:37.389 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:08:37.389 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:37.389 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:37.390 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:08:37.390 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:37.390 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:08:37.390 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:37.391 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:37.391 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:08:37.391 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:37.391 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:08:37.391 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:37.392 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:08:37.392 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:37.392 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:37.392 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:37.393 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:37.393 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:08:37.393 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:37.393 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:37.393 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:08:37.394 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:37.394 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:37.394 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:37.394 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:08:37.395 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:37.395 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:37.395 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:37.396 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:37.396 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:08:37.396 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:37.397 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:37.397 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.957515_216.1", 2021-09-16T20:08:37.397 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:37.397 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:37.398 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:37.398 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.957515Z", 2021-09-16T20:08:37.398 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:37.398 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:37.399 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:37.399 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:37.399 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:08:37.399 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:37.400 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:37.400 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:08:37.400 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:37.400 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:37.400 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:37.401 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:08:37.401 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:37.401 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:37.401 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:37.401 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:08:37.402 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:37.402 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:08:37.402 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:08:37.402 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:08:37.403 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:37.403 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:08:37.404 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.957515_216.1 2021-09-16T20:08:42.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:42.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:42.407 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:08:42.455 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:08:42.456 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:08:42.542 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:08:42.542 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:08:42.543 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:08:42.543 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:08:42.543 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:08:42.543 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:08:42.543 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:42.544 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:42.544 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:08:42.544 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:42.544 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:08:42.545 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:42.545 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:42.545 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:08:42.545 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:42.545 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:08:42.546 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:42.546 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:08:42.546 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:42.546 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:42.547 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:42.547 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:42.547 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:08:42.547 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:42.548 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:42.548 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:08:42.548 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:42.548 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:42.549 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:42.549 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:08:42.549 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:42.549 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:42.549 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:42.550 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:42.550 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:08:42.550 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:42.550 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:42.551 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.788603_210.1", 2021-09-16T20:08:42.551 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:42.551 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:42.551 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:42.552 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.788603Z", 2021-09-16T20:08:42.552 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:42.552 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:42.552 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:42.553 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:42.553 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:08:42.553 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:42.553 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:42.553 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:08:42.554 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:42.554 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:42.554 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:42.554 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:08:42.555 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:42.555 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:42.555 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:42.555 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:08:42.555 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:42.556 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:08:42.556 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:08:42.556 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:08:42.556 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:42.556 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:08:42.558 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.788603_210.1 2021-09-16T20:08:47.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:47.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:47.560 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:08:47.651 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:08:47.651 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:08:47.770 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:08:47.770 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:08:47.770 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:08:47.771 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:08:47.771 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:08:47.771 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:08:47.771 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:47.772 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:47.772 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:08:47.772 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:47.772 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:08:47.772 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:47.773 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:47.773 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:08:47.773 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:47.773 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:08:47.774 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:47.774 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:08:47.774 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:47.774 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:47.774 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:47.775 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:47.775 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:08:47.775 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:47.775 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:47.775 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:08:47.776 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:47.776 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:47.776 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:47.776 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:08:47.777 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:47.777 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:47.777 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:47.777 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:47.777 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:08:47.778 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:47.778 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:47.778 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.765733_208.1", 2021-09-16T20:08:47.778 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:47.779 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:47.779 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:47.779 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.765733Z", 2021-09-16T20:08:47.779 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:47.780 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:47.780 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:47.780 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:47.780 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:08:47.780 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:47.781 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:47.781 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:08:47.781 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:47.781 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:47.781 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:47.782 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:08:47.782 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:47.782 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:47.782 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:47.782 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:08:47.783 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:47.783 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:08:47.783 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:08:47.783 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:08:47.784 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:47.784 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:08:47.785 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.765733_208.1 2021-09-16T20:08:52.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:52.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:52.788 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:08:52.856 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:08:52.856 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:08:53.071 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:08:53.072 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:08:53.072 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:08:53.072 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:08:53.072 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:08:53.073 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:08:53.073 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:53.073 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:53.073 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:08:53.073 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:53.074 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:08:53.074 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:53.074 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:53.074 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:08:53.075 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:53.075 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:08:53.075 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:53.075 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:08:53.076 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:53.076 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:53.076 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:53.076 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:53.077 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:08:53.077 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:53.077 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:53.077 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:08:53.077 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:53.078 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:53.078 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:53.078 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:08:53.078 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:53.078 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:53.079 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:53.079 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:53.079 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:08:53.079 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:53.080 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:53.080 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.957515_216.1", 2021-09-16T20:08:53.080 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:53.081 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:53.081 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:53.081 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.957515Z", 2021-09-16T20:08:53.081 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:53.081 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:53.082 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:53.082 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:53.082 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:08:53.082 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:53.083 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:53.083 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:08:53.083 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:53.083 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:53.083 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:53.084 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:08:53.084 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:53.084 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:53.084 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:53.084 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:08:53.085 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:53.085 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:08:53.085 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:08:53.085 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:08:53.085 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:53.086 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:08:53.087 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.957515_216.1 2021-09-16T20:08:58.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:58.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:08:58.089 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:08:58.131 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:08:58.132 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:08:58.197 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:08:58.197 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:08:58.197 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:08:58.198 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:08:58.198 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:08:58.198 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:08:58.198 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:58.199 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:58.199 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:08:58.199 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:58.199 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:08:58.199 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:58.200 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:58.200 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:08:58.200 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:58.200 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:08:58.200 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:58.201 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:08:58.201 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:58.201 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:58.201 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:58.202 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:58.202 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:08:58.202 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:58.202 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:58.203 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:08:58.203 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:58.203 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:58.203 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:58.203 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:08:58.204 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:58.204 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:58.204 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:58.204 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:58.205 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:08:58.205 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:58.205 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:58.205 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.765733_208.1", 2021-09-16T20:08:58.205 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:58.206 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:58.206 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:58.206 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.765733Z", 2021-09-16T20:08:58.206 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:58.206 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:58.207 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:58.207 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:08:58.207 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:08:58.207 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:08:58.208 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:08:58.208 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:08:58.208 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:08:58.208 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:08:58.209 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:08:58.209 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:08:58.209 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:08:58.209 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:58.209 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:58.210 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:08:58.210 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:08:58.210 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:08:58.210 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:08:58.210 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:08:58.211 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:08:58.211 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:08:58.211 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.765733_208.1 2021-09-16T20:09:03.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:03.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:03.214 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:09:03.264 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:09:03.264 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:09:03.322 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:09:03.322 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:09:03.323 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:09:03.323 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:09:03.323 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:09:03.323 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:09:03.324 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:03.324 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:03.324 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:09:03.324 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:03.325 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:09:03.325 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:03.325 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:03.325 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:09:03.325 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:03.326 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:09:03.326 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:03.326 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:09:03.326 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:03.327 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:03.327 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:03.327 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:03.327 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:09:03.327 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:03.328 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:03.328 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:09:03.328 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:03.328 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:03.329 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:03.329 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:09:03.329 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:03.329 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:03.329 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:03.330 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:03.330 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:09:03.330 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:03.330 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:03.331 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.765733_208.1", 2021-09-16T20:09:03.331 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:03.331 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:03.331 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:03.331 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.765733Z", 2021-09-16T20:09:03.332 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:03.332 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:03.332 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:03.332 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:03.333 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:09:03.333 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:03.333 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:03.333 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:09:03.333 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:03.334 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:03.334 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:03.334 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:09:03.334 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:03.335 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:03.335 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:03.335 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:09:03.335 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:03.336 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:09:03.336 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:09:03.336 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:09:03.336 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:03.336 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:09:03.338 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.765733_208.1 2021-09-16T20:09:08.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:08.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:08.340 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:09:08.392 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:09:08.392 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:09:08.656 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:09:08.656 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:09:08.656 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:09:08.656 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:09:08.657 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:09:08.657 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:09:08.657 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:08.657 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:08.658 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:09:08.658 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:08.658 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:09:08.658 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:08.659 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:08.659 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:09:08.659 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:08.659 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:09:08.659 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:08.660 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:09:08.660 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:08.660 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:08.660 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:08.660 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:08.661 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:09:08.661 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:08.661 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:08.661 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:09:08.662 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:08.662 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:08.662 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:08.662 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:09:08.663 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:08.663 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:08.663 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:08.663 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:08.663 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:09:08.664 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:08.664 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:08.664 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.895465_215.1", 2021-09-16T20:09:08.664 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:08.665 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:08.665 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:08.665 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.895465Z", 2021-09-16T20:09:08.665 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:08.666 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:08.666 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:08.666 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:08.666 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:09:08.666 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:08.667 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:08.667 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:09:08.667 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:08.667 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:08.668 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:08.668 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:09:08.668 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:08.668 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:08.668 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:08.669 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:09:08.669 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:08.669 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:09:08.669 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:09:08.669 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:09:08.670 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:08.670 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:09:08.670 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.895465_215.1 2021-09-16T20:09:13.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:13.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:13.673 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:09:13.729 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:09:13.729 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:09:13.853 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:09:13.853 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:09:13.854 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:09:13.854 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:09:13.854 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:09:13.855 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:09:13.855 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:13.855 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:13.855 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:09:13.856 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:13.856 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:09:13.856 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:13.856 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:13.856 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:09:13.857 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:13.857 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:09:13.857 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:13.858 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:09:13.858 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:13.858 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:13.858 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:13.859 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:13.859 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:09:13.859 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:13.859 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:13.859 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:09:13.860 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:13.860 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:13.860 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:13.861 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:09:13.861 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:13.861 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:13.861 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:13.861 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:13.862 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:09:13.862 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:13.862 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:13.862 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.765733_208.1", 2021-09-16T20:09:13.863 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:13.863 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:13.863 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:13.863 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.765733Z", 2021-09-16T20:09:13.863 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:13.864 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:13.864 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:13.864 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:13.864 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:09:13.865 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:13.865 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:13.865 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:09:13.865 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:13.865 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:13.866 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:13.866 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:09:13.866 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:13.866 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:13.867 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:13.867 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:09:13.867 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:13.867 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:09:13.868 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:09:13.868 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:09:13.868 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:13.868 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:09:13.869 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.765733_208.1 2021-09-16T20:09:18.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:18.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:18.872 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:09:18.914 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:09:18.915 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:09:18.975 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:09:18.975 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:09:18.975 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:09:18.976 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:09:18.976 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:09:18.976 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:09:18.977 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:18.977 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:18.977 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:09:18.977 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:18.978 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:09:18.978 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:18.978 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:18.978 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:09:18.979 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:18.979 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:09:18.979 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:18.980 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:09:18.980 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:18.980 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:18.980 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:18.981 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:18.981 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:09:18.981 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:18.981 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:18.982 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:09:18.982 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:18.982 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:18.982 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:18.983 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:09:18.983 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:18.983 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:18.983 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:18.984 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:18.984 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:09:18.984 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:18.984 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:18.985 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.765733_208.1", 2021-09-16T20:09:18.985 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:18.985 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:18.985 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:18.986 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.765733Z", 2021-09-16T20:09:18.986 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:18.986 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:18.986 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:18.987 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:18.987 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:09:18.987 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:18.987 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:18.988 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:09:18.988 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:18.988 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:18.988 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:18.989 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:09:18.989 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:18.989 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:18.989 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:18.990 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:09:18.990 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:18.990 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:09:18.990 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:09:18.991 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:09:18.991 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:18.991 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:09:18.992 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.765733_208.1 2021-09-16T20:09:23.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:23.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:23.995 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:09:24.041 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:09:24.042 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:09:24.112 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:09:24.113 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:09:24.113 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:09:24.113 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:09:24.114 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:09:24.114 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:09:24.114 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:24.114 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:24.115 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:09:24.115 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:24.115 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:09:24.115 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:24.116 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:24.116 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:09:24.116 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:24.116 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:09:24.116 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:24.117 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:09:24.117 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:24.117 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:24.117 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:24.117 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:24.118 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:09:24.118 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:24.118 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:24.118 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:09:24.119 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:24.119 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:24.119 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:24.119 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:09:24.120 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:24.120 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:24.120 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:24.120 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:24.120 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:09:24.121 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:24.121 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:24.121 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:09:24.121 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:24.122 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:24.122 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:24.122 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:09:24.122 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:24.123 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:24.123 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:24.123 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:24.123 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:09:24.124 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:24.124 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:24.124 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:09:24.124 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:24.125 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:24.125 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:24.125 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:09:24.125 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:24.125 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:24.126 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:24.126 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:09:24.126 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:24.126 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:09:24.127 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:09:24.127 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:09:24.127 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:24.127 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:09:24.129 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:09:29.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:29.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:29.131 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:09:29.180 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:09:29.180 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:09:29.246 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:09:29.246 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:09:29.246 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:09:29.247 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:09:29.247 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:09:29.247 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:09:29.247 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:29.248 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:29.248 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:09:29.248 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:29.248 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:09:29.249 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:29.249 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:29.249 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:09:29.249 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:29.250 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:09:29.250 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:29.250 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:09:29.251 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:29.251 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:29.251 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:29.251 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:29.252 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:09:29.252 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:29.252 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:29.252 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:09:29.252 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:29.253 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:29.253 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:29.253 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:09:29.253 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:29.253 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:29.254 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:29.254 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:29.254 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:09:29.255 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:29.255 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:29.255 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.957515_216.1", 2021-09-16T20:09:29.255 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:29.256 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:29.256 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:29.256 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.957515Z", 2021-09-16T20:09:29.256 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:29.257 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:29.257 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:29.257 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:29.257 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:09:29.257 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:29.258 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:29.258 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:09:29.258 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:29.258 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:29.258 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:29.259 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:09:29.259 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:29.259 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:29.259 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:29.260 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:09:29.260 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:29.260 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:09:29.260 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:09:29.261 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:09:29.261 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:29.261 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:09:29.263 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.957515_216.1 2021-09-16T20:09:34.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:34.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:34.266 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:09:34.346 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:09:34.347 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:09:34.548 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:09:34.548 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:09:34.548 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:09:34.549 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:09:34.549 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:09:34.549 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:09:34.549 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:34.549 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:34.550 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:09:34.550 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:34.550 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:09:34.550 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:34.550 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:34.551 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:09:34.551 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:34.551 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:09:34.551 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:34.552 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:09:34.552 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:34.552 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:34.552 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:34.552 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:34.553 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:09:34.553 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:34.553 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:34.553 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:09:34.554 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:34.554 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:34.554 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:34.554 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:09:34.554 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:34.555 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:34.555 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:34.555 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:34.555 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:09:34.556 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:34.556 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:34.556 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.765733_208.1", 2021-09-16T20:09:34.556 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:34.556 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:34.557 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:34.557 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.765733Z", 2021-09-16T20:09:34.557 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:34.557 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:34.557 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:34.558 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:34.558 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:09:34.558 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:34.558 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:34.558 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:09:34.559 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:34.559 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:34.559 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:34.559 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:09:34.559 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:34.560 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:34.560 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:34.560 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:09:34.560 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:34.560 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:09:34.561 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:09:34.561 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:09:34.561 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:34.561 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:09:34.562 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.765733_208.1 2021-09-16T20:09:39.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:39.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:39.565 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:09:39.698 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:09:39.699 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:09:39.753 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:09:39.754 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:09:39.754 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:09:39.754 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:09:39.755 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:09:39.755 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:09:39.755 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:39.755 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:39.755 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:09:39.756 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:39.756 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:09:39.756 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:39.756 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:39.757 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:09:39.757 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:39.757 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:09:39.757 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:39.757 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:09:39.758 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:39.758 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:39.758 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:39.758 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:39.759 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:09:39.759 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:39.759 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:39.760 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:09:39.760 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:39.760 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:39.760 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:39.761 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:09:39.761 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:39.761 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:39.762 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:39.762 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:39.762 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:09:39.763 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:39.763 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:39.764 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.768089_209.1", 2021-09-16T20:09:39.764 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:39.765 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:39.765 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:39.765 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.768089Z", 2021-09-16T20:09:39.766 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:39.766 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:39.767 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:39.767 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:39.768 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:09:39.768 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:39.768 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:39.769 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:09:39.769 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:39.770 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:39.770 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:39.770 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:09:39.771 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:39.771 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:39.772 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:39.772 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:09:39.773 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:39.773 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:09:39.773 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:09:39.774 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:09:39.774 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:39.775 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:09:39.777 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.768089_209.1 2021-09-16T20:09:44.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:44.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:44.780 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:09:44.894 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:09:44.895 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:09:44.960 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:09:44.960 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:09:44.960 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:09:44.960 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:09:44.961 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:09:44.961 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:09:44.961 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:44.961 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:44.961 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:09:44.962 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:44.962 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:09:44.962 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:44.962 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:44.963 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:09:44.963 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:44.963 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:09:44.963 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:44.964 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:09:44.964 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:44.964 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:44.964 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:44.964 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:44.965 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:09:44.965 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:44.965 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:44.965 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:09:44.966 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:44.966 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:44.966 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:44.966 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:09:44.966 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:44.967 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:44.967 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:44.967 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:44.967 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:09:44.968 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:44.968 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:44.968 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.788603_210.1", 2021-09-16T20:09:44.968 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:44.968 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:44.969 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:44.969 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.788603Z", 2021-09-16T20:09:44.969 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:44.969 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:44.970 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:44.970 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:44.970 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:09:44.970 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:44.970 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:44.971 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:09:44.971 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:44.971 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:44.971 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:44.971 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:09:44.972 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:44.972 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:44.972 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:44.972 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:09:44.973 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:44.973 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:09:44.973 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:09:44.973 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:09:44.973 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:44.974 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:09:44.975 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.788603_210.1 2021-09-16T20:09:49.977 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:49.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:49.978 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:09:50.026 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:09:50.026 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:09:50.113 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:09:50.113 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:09:50.113 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:09:50.114 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:09:50.114 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:09:50.114 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:09:50.114 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:50.115 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:50.115 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:09:50.115 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:50.115 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:09:50.115 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:50.116 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:50.116 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:09:50.116 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:50.116 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:09:50.116 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:50.117 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:09:50.117 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:50.117 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:50.117 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:50.118 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:50.118 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:09:50.118 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:50.118 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:50.119 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:09:50.119 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:50.119 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:50.119 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:50.120 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:09:50.120 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:50.120 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:50.120 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:50.121 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:50.121 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:09:50.121 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:50.121 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:50.121 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.764797_207.1", 2021-09-16T20:09:50.122 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:50.122 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:50.122 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:50.122 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.764797Z", 2021-09-16T20:09:50.122 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:50.123 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:50.123 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:50.123 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:50.123 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:09:50.124 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:50.124 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:50.124 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:09:50.124 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:50.124 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:50.125 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:50.125 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:09:50.125 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:50.125 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:50.125 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:50.126 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:09:50.126 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:50.126 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:09:50.126 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:09:50.127 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:09:50.127 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:50.127 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:09:50.128 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.764797_207.1 2021-09-16T20:09:55.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:55.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:09:55.131 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:09:55.183 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:09:55.183 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:09:55.486 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:09:55.487 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:09:55.487 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:09:55.487 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:09:55.487 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:09:55.488 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:09:55.488 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:55.488 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:55.488 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:09:55.488 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:55.489 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:09:55.489 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:55.489 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:55.489 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:09:55.490 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:55.490 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:09:55.490 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:55.490 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:09:55.490 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:55.491 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:55.491 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:55.491 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:55.491 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:09:55.492 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:55.492 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:55.492 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:09:55.492 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:55.492 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:55.493 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:55.493 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:09:55.493 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:55.493 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:55.494 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:55.494 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:55.495 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:09:55.495 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:55.495 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:55.495 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.891592_214.1", 2021-09-16T20:09:55.496 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:55.496 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:55.496 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:55.496 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.891592Z", 2021-09-16T20:09:55.497 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:55.497 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:55.497 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:55.497 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:09:55.497 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:09:55.498 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:09:55.498 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:09:55.498 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:09:55.498 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:09:55.499 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:09:55.499 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:09:55.499 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:09:55.499 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:09:55.499 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:55.500 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:55.500 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:09:55.500 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:09:55.500 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:09:55.501 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:09:55.501 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:09:55.501 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:09:55.501 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:09:55.502 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.891592_214.1 2021-09-16T20:10:00.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:00.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:00.504 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:10:00.629 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:10:00.630 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:10:00.765 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:10:00.766 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:10:00.766 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:10:00.766 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:10:00.766 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:10:00.767 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:10:00.767 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:00.767 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:00.767 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:10:00.768 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:00.768 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:10:00.768 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:00.768 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:00.768 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:10:00.769 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:00.769 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:10:00.769 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:00.769 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:10:00.769 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:00.770 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:00.770 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:00.770 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:00.770 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:10:00.771 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:00.771 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:00.771 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:10:00.771 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:00.771 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:00.772 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:00.772 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:10:00.772 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:00.772 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:00.772 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:00.773 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:00.773 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:10:00.773 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:00.773 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:00.774 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.971080_217.1", 2021-09-16T20:10:00.774 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:00.774 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:00.774 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:00.774 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.971080Z", 2021-09-16T20:10:00.775 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:00.775 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:00.775 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:00.775 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:00.776 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:10:00.776 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:00.776 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:00.776 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:10:00.776 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:00.777 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:00.777 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:00.777 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:10:00.777 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:00.778 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:00.778 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:00.778 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:10:00.778 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:00.779 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:10:00.779 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:10:00.779 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:10:00.779 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:00.779 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:10:00.781 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.971080_217.1 2021-09-16T20:10:05.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:05.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:05.784 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:10:05.826 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:10:05.826 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:10:05.893 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:10:05.893 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:10:05.894 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:10:05.894 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:10:05.894 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:10:05.894 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:10:05.895 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:05.895 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:05.895 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:10:05.895 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:05.895 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:10:05.896 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:05.896 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:05.896 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:10:05.896 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:05.897 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:10:05.897 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:05.897 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:10:05.897 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:05.898 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:05.898 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:05.898 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:05.898 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:10:05.898 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:05.899 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:05.899 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:10:05.899 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:05.899 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:05.899 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:05.900 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:10:05.900 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:05.900 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:05.900 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:05.901 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:05.901 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:10:05.901 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:05.901 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:05.901 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.895465_215.1", 2021-09-16T20:10:05.902 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:05.902 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:05.902 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:05.902 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.895465Z", 2021-09-16T20:10:05.903 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:05.903 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:05.903 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:05.903 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:05.903 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:10:05.904 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:05.904 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:05.904 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:10:05.904 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:05.904 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:05.905 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:05.905 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:10:05.905 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:05.905 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:05.906 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:05.906 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:10:05.906 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:05.906 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:10:05.906 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:10:05.907 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:10:05.907 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:05.907 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:10:05.908 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.895465_215.1 2021-09-16T20:10:10.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:10.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:10.911 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:10:10.958 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:10:10.958 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:10:11.023 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:10:11.024 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:10:11.024 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:10:11.024 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:10:11.024 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:10:11.025 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:10:11.025 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:11.025 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:11.025 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:10:11.026 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:11.026 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:10:11.026 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:11.026 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:11.027 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:10:11.027 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:11.027 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:10:11.027 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:11.028 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:10:11.028 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:11.028 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:11.028 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:11.029 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:11.029 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:10:11.029 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:11.029 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:11.029 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:10:11.030 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:11.030 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:11.030 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:11.030 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:10:11.031 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:11.031 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:11.031 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:11.031 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:11.032 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:10:11.032 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:11.032 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:11.032 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.957515_216.1", 2021-09-16T20:10:11.032 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:11.033 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:11.033 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:11.033 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.957515Z", 2021-09-16T20:10:11.033 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:11.034 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:11.034 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:11.035 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:11.035 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:10:11.035 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:11.035 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:11.036 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:10:11.036 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:11.036 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:11.036 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:11.036 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:10:11.037 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:11.037 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:11.037 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:11.037 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:10:11.038 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:11.038 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:10:11.038 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:10:11.038 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:10:11.038 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:11.039 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:10:11.040 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.957515_216.1 2021-09-16T20:10:16.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:16.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:16.042 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:10:16.106 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:10:16.106 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:10:16.314 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:10:16.314 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:10:16.314 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:10:16.315 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:10:16.315 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:10:16.315 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:10:16.315 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:16.315 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:16.316 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:10:16.316 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:16.316 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:10:16.316 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:16.316 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:16.317 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:10:16.317 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:16.317 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:10:16.317 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:16.318 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:10:16.318 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:16.318 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:16.318 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:16.318 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:16.319 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:10:16.319 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:16.319 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:16.319 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:10:16.319 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:16.320 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:16.320 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:16.320 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:10:16.320 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:16.321 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:16.321 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:16.321 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:16.321 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:10:16.321 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:16.322 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:16.322 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.764797_207.1", 2021-09-16T20:10:16.322 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:16.322 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:16.322 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:16.323 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.764797Z", 2021-09-16T20:10:16.323 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:16.323 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:16.323 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:16.324 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:16.324 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:10:16.324 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:16.324 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:16.324 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:10:16.325 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:16.325 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:16.325 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:16.325 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:10:16.326 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:16.326 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:16.326 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:16.326 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:10:16.326 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:16.327 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:10:16.327 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:10:16.327 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:10:16.327 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:16.327 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:10:16.328 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.764797_207.1 2021-09-16T20:10:21.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:21.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:21.331 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:10:21.531 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:10:21.531 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:10:21.660 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:10:21.660 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:10:21.660 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:10:21.661 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:10:21.661 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:10:21.661 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:10:21.661 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:21.662 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:21.662 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:10:21.662 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:21.663 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:10:21.663 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:21.663 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:21.663 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:10:21.663 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:21.664 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:10:21.664 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:21.664 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:10:21.664 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:21.665 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:21.665 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:21.665 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:21.665 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:10:21.665 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:21.666 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:21.666 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:10:21.666 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:21.666 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:21.666 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:21.667 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:10:21.667 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:21.667 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:21.667 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:21.668 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:21.668 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:10:21.668 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:21.668 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:21.668 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.841894_213.1", 2021-09-16T20:10:21.669 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:21.669 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:21.669 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:21.669 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.841894Z", 2021-09-16T20:10:21.670 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:21.670 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:21.670 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:21.670 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:21.670 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:10:21.671 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:21.671 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:21.671 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:10:21.671 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:21.671 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:21.672 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:21.672 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:10:21.672 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:21.673 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:21.673 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:21.673 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:10:21.673 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:21.674 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:10:21.674 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:10:21.674 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:10:21.675 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:21.675 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:10:21.676 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.841894_213.1 2021-09-16T20:10:26.677 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:26.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:26.679 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:10:26.723 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:10:26.723 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:10:26.788 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:10:26.788 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:10:26.788 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:10:26.789 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:10:26.789 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:10:26.789 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:10:26.789 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:26.790 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:26.790 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:10:26.790 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:26.790 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:10:26.791 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:26.791 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:26.791 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:10:26.791 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:26.792 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:10:26.792 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:26.792 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:10:26.792 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:26.792 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:26.793 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:26.793 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:26.793 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:10:26.793 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:26.793 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:26.794 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:10:26.794 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:26.794 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:26.794 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:26.795 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:10:26.795 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:26.795 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:26.795 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:26.796 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:26.796 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:10:26.796 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:26.796 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:26.796 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.765733_208.1", 2021-09-16T20:10:26.797 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:26.797 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:26.797 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:26.797 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.765733Z", 2021-09-16T20:10:26.798 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:26.798 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:26.798 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:26.798 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:26.799 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:10:26.799 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:26.799 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:26.799 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:10:26.800 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:26.800 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:26.800 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:26.800 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:10:26.801 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:26.801 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:26.801 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:26.801 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:10:26.802 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:26.802 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:10:26.802 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:10:26.802 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:10:26.803 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:26.803 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:10:26.804 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.765733_208.1 2021-09-16T20:10:31.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:31.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:31.807 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:10:31.854 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:10:31.855 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:10:31.921 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:10:31.921 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:10:31.922 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:10:31.922 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:10:31.922 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:10:31.922 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:10:31.922 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:31.923 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:31.923 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:10:31.923 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:31.923 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:10:31.924 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:31.924 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:31.924 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:10:31.924 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:31.924 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:10:31.925 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:31.925 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:10:31.925 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:31.925 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:31.926 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:31.926 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:31.926 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:10:31.926 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:31.926 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:31.927 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:10:31.927 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:31.927 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:31.927 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:31.928 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:10:31.928 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:31.928 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:31.928 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:31.928 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:31.929 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:10:31.929 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:31.929 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:31.929 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821521.391283_240.1", 2021-09-16T20:10:31.929 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:31.930 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:31.930 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:31.930 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:21.391283Z", 2021-09-16T20:10:31.930 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:31.930 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:31.931 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:31.931 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:31.931 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:10:31.931 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:31.932 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:31.932 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:10:31.932 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:31.932 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:31.932 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:31.933 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:10:31.933 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:31.933 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:31.933 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:31.933 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:10:31.934 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:31.934 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:10:31.934 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:10:31.934 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:10:31.935 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:31.935 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:10:31.936 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821521.391283_240.1 2021-09-16T20:10:36.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:36.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:36.939 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:10:37.085 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:10:37.086 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:10:37.148 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:10:37.149 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:10:37.149 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:10:37.149 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:10:37.150 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:10:37.150 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:10:37.150 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:37.150 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:37.151 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:10:37.151 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:37.151 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:10:37.152 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:37.152 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:37.152 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:10:37.152 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:37.153 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:10:37.153 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:37.153 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:10:37.153 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:37.153 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:37.154 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:37.154 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:37.154 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:10:37.154 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:37.155 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:37.155 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:10:37.155 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:37.155 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:37.156 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:37.156 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:10:37.156 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:37.156 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:37.156 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:37.157 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:37.157 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:10:37.157 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:37.157 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:37.158 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.768089_209.1", 2021-09-16T20:10:37.158 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:37.158 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:37.158 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:37.158 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.768089Z", 2021-09-16T20:10:37.159 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:37.159 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:37.159 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:37.159 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:37.160 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:10:37.160 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:37.160 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:37.160 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:10:37.160 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:37.161 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:37.161 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:37.161 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:10:37.161 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:37.162 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:37.162 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:37.162 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:10:37.162 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:37.162 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:10:37.163 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:10:37.163 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:10:37.163 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:37.163 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:10:37.165 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.768089_209.1 2021-09-16T20:10:42.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:42.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:42.167 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:10:42.222 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:10:42.222 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:10:42.411 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:10:42.411 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:10:42.412 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:10:42.412 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:10:42.412 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:10:42.412 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:10:42.412 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:42.413 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:42.413 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:10:42.413 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:42.413 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:10:42.414 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:42.414 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:42.414 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:10:42.414 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:42.414 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:10:42.415 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:42.415 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:10:42.415 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:42.415 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:42.416 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:42.416 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:42.416 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:10:42.416 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:42.416 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:42.416 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:10:42.417 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:42.417 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:42.417 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:42.417 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:10:42.418 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:42.418 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:42.418 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:42.418 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:42.418 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:10:42.419 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:42.419 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:42.419 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821519.116095_226.1", 2021-09-16T20:10:42.419 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:42.419 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:42.419 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:42.420 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:19.116095Z", 2021-09-16T20:10:42.420 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:42.420 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:42.420 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:42.420 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:42.421 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:10:42.421 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:42.421 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:42.421 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:10:42.421 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:42.422 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:42.422 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:42.422 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:10:42.422 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:42.423 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:42.423 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:42.423 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:10:42.423 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:42.423 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:10:42.424 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:10:42.424 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:10:42.424 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:42.424 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:10:42.425 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821519.116095_226.1 2021-09-16T20:10:47.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:47.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:47.427 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:10:47.485 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:10:47.486 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:10:47.685 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:10:47.685 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:10:47.685 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:10:47.686 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:10:47.686 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:10:47.686 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:10:47.686 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:47.687 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:47.687 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:10:47.687 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:47.687 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:10:47.688 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:47.688 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:47.688 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:10:47.688 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:47.688 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:10:47.689 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:47.689 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:10:47.689 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:47.689 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:47.689 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:47.690 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:47.690 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:10:47.690 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:47.690 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:47.691 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:10:47.691 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:47.691 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:47.691 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:47.691 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:10:47.692 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:47.692 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:47.692 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:47.692 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:47.693 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:10:47.693 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:47.693 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:47.693 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.957515_216.1", 2021-09-16T20:10:47.693 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:47.694 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:47.694 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:47.694 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.957515Z", 2021-09-16T20:10:47.694 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:47.695 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:47.695 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:47.695 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:47.695 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:10:47.695 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:47.696 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:47.696 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:10:47.696 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:47.696 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:47.696 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:47.697 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:10:47.697 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:47.697 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:47.697 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:47.698 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:10:47.698 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:47.698 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:10:47.698 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:10:47.699 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:10:47.699 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:47.699 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:10:47.700 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.957515_216.1 2021-09-16T20:10:52.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:52.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:52.702 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:10:52.745 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:10:52.745 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:10:52.807 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:10:52.808 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:10:52.808 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:10:52.808 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:10:52.808 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:10:52.809 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:10:52.809 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:52.809 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:52.809 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:10:52.810 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:52.810 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:10:52.810 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:52.810 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:52.811 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:10:52.811 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:52.811 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:10:52.811 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:52.811 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:10:52.812 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:52.812 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:52.812 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:52.812 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:52.813 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:10:52.813 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:52.813 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:52.813 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:10:52.814 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:52.814 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:52.814 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:52.814 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:10:52.815 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:52.815 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:52.815 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:52.815 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:52.815 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:10:52.816 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:52.816 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:52.816 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821519.046948_223.1", 2021-09-16T20:10:52.816 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:52.816 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:52.817 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:52.817 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:19.046948Z", 2021-09-16T20:10:52.817 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:52.817 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:52.817 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:52.818 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:52.818 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:10:52.818 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:52.818 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:52.819 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:10:52.819 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:52.819 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:52.819 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:52.820 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:10:52.820 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:52.820 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:52.820 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:52.820 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:10:52.821 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:52.821 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:10:52.821 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:10:52.821 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:10:52.822 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:52.822 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:10:52.823 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821519.046948_223.1 2021-09-16T20:10:57.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:57.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:10:57.827 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:10:57.876 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:10:57.877 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:10:57.943 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:10:57.943 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:10:57.944 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:10:57.944 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:10:57.944 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:10:57.945 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:10:57.945 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:57.945 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:57.945 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:10:57.946 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:57.946 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:10:57.946 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:57.946 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:57.947 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:10:57.947 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:57.947 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:10:57.947 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:57.948 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:10:57.948 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:57.948 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:57.948 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:57.948 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:57.949 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:10:57.949 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:57.949 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:57.949 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:10:57.950 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:57.950 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:57.950 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:57.950 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:10:57.950 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:57.951 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:57.951 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:57.951 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:57.951 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:10:57.952 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:57.952 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:57.952 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821833.716837_266.1", 2021-09-16T20:10:57.952 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:57.953 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:57.953 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:57.953 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:50:33.716837Z", 2021-09-16T20:10:57.953 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:57.954 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:57.954 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:57.954 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:10:57.954 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:10:57.954 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:10:57.955 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:10:57.955 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:10:57.955 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:10:57.955 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:10:57.956 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:10:57.956 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:10:57.956 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:10:57.956 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:57.956 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:57.957 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:10:57.957 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:10:57.957 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:10:57.957 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:10:57.958 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:10:57.958 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:10:57.958 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:10:57.959 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821833.716837_266.1 2021-09-16T20:11:02.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:02.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:02.962 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:11:03.024 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:11:03.024 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:11:03.429 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:11:03.429 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:11:03.430 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:11:03.430 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:11:03.430 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:11:03.430 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:11:03.431 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:03.431 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:03.431 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:11:03.431 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:03.432 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:11:03.432 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:03.432 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:03.432 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:11:03.432 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:03.433 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:11:03.433 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:03.433 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:11:03.433 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:03.434 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:03.434 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:03.434 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:03.434 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:11:03.434 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:03.435 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:03.435 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:11:03.435 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:03.435 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:03.435 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:03.436 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:11:03.436 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:03.436 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:03.436 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:03.437 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:03.437 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:11:03.437 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:03.437 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:03.438 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.791325_211.1", 2021-09-16T20:11:03.438 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:03.438 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:03.439 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:03.439 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.791325Z", 2021-09-16T20:11:03.439 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:03.439 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:03.439 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:03.440 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:03.440 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:11:03.440 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:03.440 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:03.440 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:11:03.441 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:03.441 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:03.441 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:03.441 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:11:03.441 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:03.442 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:03.442 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:03.442 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:11:03.442 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:03.443 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:11:03.443 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:11:03.443 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:11:03.443 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:03.443 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:11:03.444 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.791325_211.1 2021-09-16T20:11:08.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:08.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:08.447 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:11:08.505 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:11:08.505 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:11:08.599 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:11:08.599 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:11:08.600 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:11:08.600 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:11:08.600 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:11:08.600 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:11:08.601 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:08.601 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:08.601 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:11:08.601 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:08.602 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:11:08.602 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:08.602 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:08.602 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:11:08.602 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:08.603 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:11:08.603 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:08.603 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:11:08.603 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:08.604 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:08.604 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:08.604 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:08.604 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:11:08.605 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:08.605 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:08.605 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:11:08.605 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:08.605 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:08.606 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:08.606 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:11:08.606 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:08.606 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:08.607 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:08.607 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:08.607 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:11:08.607 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:08.607 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:08.608 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821519.032233_221.1", 2021-09-16T20:11:08.608 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:08.608 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:08.608 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:08.609 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:19.032233Z", 2021-09-16T20:11:08.609 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:08.609 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:08.609 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:08.609 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:08.610 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:11:08.610 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:08.610 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:08.610 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:11:08.611 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:08.611 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:08.611 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:08.611 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:11:08.612 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:08.612 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:08.612 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:08.612 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:11:08.612 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:08.613 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:11:08.613 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:11:08.613 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:11:08.613 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:08.613 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:11:08.615 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821519.032233_221.1 2021-09-16T20:11:13.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:13.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:13.617 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:11:13.663 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:11:13.663 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:11:13.730 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:11:13.730 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:11:13.731 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:11:13.731 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:11:13.731 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:11:13.731 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:11:13.732 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:13.732 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:13.732 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:11:13.732 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:13.733 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:11:13.733 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:13.733 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:13.734 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:11:13.734 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:13.734 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:11:13.734 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:13.735 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:11:13.735 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:13.735 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:13.735 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:13.736 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:13.736 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:11:13.736 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:13.736 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:13.736 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:11:13.737 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:13.737 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:13.737 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:13.737 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:11:13.738 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:13.738 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:13.738 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:13.738 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:13.738 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:11:13.739 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:13.739 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:13.739 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.839033_212.1", 2021-09-16T20:11:13.739 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:13.740 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:13.740 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:13.740 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.839033Z", 2021-09-16T20:11:13.740 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:13.741 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:13.741 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:13.741 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:13.741 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:11:13.741 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:13.742 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:13.742 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:11:13.742 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:13.743 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:13.743 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:13.743 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:11:13.743 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:13.743 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:13.744 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:13.744 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:11:13.744 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:13.744 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:11:13.745 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:11:13.745 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:11:13.745 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:13.745 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:11:13.747 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.839033_212.1 2021-09-16T20:11:18.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:18.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:18.749 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:11:18.798 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:11:18.798 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:11:18.873 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:11:18.873 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:11:18.874 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:11:18.874 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:11:18.874 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:11:18.874 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:11:18.874 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:18.875 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:18.875 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:11:18.875 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:18.875 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:11:18.876 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:18.876 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:18.876 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:11:18.876 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:18.876 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:11:18.877 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:18.877 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:11:18.877 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:18.877 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:18.877 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:18.878 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:18.878 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:11:18.878 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:18.878 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:18.879 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:11:18.879 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:18.879 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:18.879 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:18.879 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:11:18.880 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:18.880 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:18.880 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:18.880 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:18.880 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:11:18.881 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:18.881 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:18.881 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.788603_210.1", 2021-09-16T20:11:18.881 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:18.881 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:18.882 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:18.882 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.788603Z", 2021-09-16T20:11:18.882 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:18.882 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:18.883 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:18.883 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:18.883 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:11:18.883 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:18.883 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:18.884 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:11:18.884 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:18.884 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:18.884 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:18.884 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:11:18.885 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:18.885 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:18.885 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:18.885 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:11:18.886 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:18.886 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:11:18.886 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:11:18.886 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:11:18.886 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:18.887 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:11:18.888 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.788603_210.1 2021-09-16T20:11:23.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:23.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:23.891 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:11:23.970 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:11:23.970 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:11:24.093 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:11:24.094 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:11:24.094 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:11:24.094 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:11:24.094 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:11:24.095 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:11:24.095 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:24.095 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:24.095 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:11:24.096 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:24.096 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:11:24.096 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:24.096 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:24.097 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:11:24.097 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:24.097 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:11:24.097 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:24.097 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:11:24.098 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:24.098 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:24.098 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:24.098 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:24.098 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:11:24.099 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:24.099 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:24.099 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:11:24.099 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:24.100 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:24.100 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:24.100 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:11:24.101 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:24.101 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:24.101 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:24.101 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:24.102 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:11:24.102 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:24.102 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:24.102 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:11:24.103 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:24.103 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:24.103 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:24.103 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:11:24.103 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:24.104 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:24.104 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:24.104 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:24.104 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:11:24.105 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:24.105 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:24.105 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:11:24.105 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:24.105 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:24.106 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:24.106 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:11:24.106 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:24.106 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:24.107 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:24.107 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:11:24.107 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:24.107 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:11:24.108 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:11:24.108 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:11:24.108 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:24.108 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:11:24.109 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:11:29.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:29.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:29.112 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:11:29.154 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:11:29.154 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:11:29.224 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:11:29.224 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:11:29.224 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:11:29.225 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:11:29.225 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:11:29.225 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:11:29.225 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:29.226 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:29.226 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:11:29.226 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:29.226 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:11:29.227 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:29.227 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:29.227 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:11:29.227 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:29.228 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:11:29.228 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:29.228 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:11:29.228 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:29.229 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:29.229 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:29.229 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:29.229 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:11:29.229 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:29.230 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:29.230 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:11:29.230 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:29.230 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:29.231 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:29.231 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:11:29.231 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:29.231 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:29.232 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:29.232 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:29.232 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:11:29.232 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:29.233 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:29.233 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.768089_209.1", 2021-09-16T20:11:29.233 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:29.233 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:29.234 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:29.234 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.768089Z", 2021-09-16T20:11:29.234 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:29.234 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:29.235 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:29.235 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:29.235 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:11:29.235 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:29.236 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:29.236 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:11:29.236 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:29.236 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:29.237 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:29.237 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:11:29.237 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:29.237 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:29.238 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:29.238 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:11:29.238 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:29.238 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:11:29.239 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:11:29.239 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:11:29.239 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:29.239 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:11:29.240 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.768089_209.1 2021-09-16T20:11:34.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:34.242 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:34.243 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:11:34.285 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:11:34.286 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:11:34.347 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:11:34.348 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:11:34.348 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:11:34.348 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:11:34.349 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:11:34.349 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:11:34.349 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:34.349 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:34.349 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:11:34.350 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:34.350 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:11:34.350 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:34.350 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:34.351 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:11:34.351 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:34.351 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:11:34.351 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:34.351 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:11:34.352 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:34.352 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:34.352 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:34.352 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:34.353 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:11:34.353 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:34.353 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:34.353 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:11:34.353 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:34.354 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:34.354 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:34.354 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:11:34.354 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:34.355 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:34.355 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:34.355 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:34.355 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:11:34.355 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:34.356 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:34.356 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821519.035503_222.1", 2021-09-16T20:11:34.356 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:34.356 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:34.356 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:34.357 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:19.035503Z", 2021-09-16T20:11:34.357 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:34.357 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:34.357 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:34.357 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:34.358 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:11:34.358 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:34.358 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:34.358 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:11:34.359 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:34.359 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:34.359 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:34.359 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:11:34.359 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:34.360 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:34.360 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:34.360 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:11:34.360 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:34.361 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:11:34.361 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:11:34.361 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:11:34.361 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:34.361 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:11:34.363 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821519.035503_222.1 2021-09-16T20:11:39.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:39.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:39.365 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:11:39.433 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:11:39.433 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:11:39.773 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:11:39.773 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:11:39.773 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:11:39.774 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:11:39.774 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:11:39.775 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:11:39.775 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:39.775 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:39.775 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:11:39.776 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:39.776 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:11:39.776 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:39.776 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:39.776 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:11:39.777 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:39.777 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:11:39.777 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:39.777 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:11:39.778 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:39.778 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:39.778 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:39.778 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:39.779 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:11:39.779 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:39.779 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:39.779 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:11:39.779 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:39.780 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:39.780 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:39.780 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:11:39.780 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:39.781 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:39.781 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:39.781 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:39.781 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:11:39.781 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:39.782 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:39.782 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.839033_212.1", 2021-09-16T20:11:39.782 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:39.782 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:39.783 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:39.783 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.839033Z", 2021-09-16T20:11:39.783 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:39.783 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:39.784 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:39.784 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:39.784 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:11:39.784 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:39.784 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:39.785 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:11:39.785 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:39.785 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:39.785 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:39.786 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:11:39.786 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:39.786 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:39.786 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:39.786 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:11:39.787 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:39.787 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:11:39.787 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:11:39.787 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:11:39.788 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:39.788 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:11:39.789 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.839033_212.1 2021-09-16T20:11:44.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:44.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:44.792 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:11:44.858 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:11:44.859 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:11:45.059 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:11:45.059 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:11:45.059 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:11:45.060 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:11:45.060 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:11:45.060 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:11:45.060 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:45.061 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:45.061 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:11:45.061 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:45.061 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:11:45.062 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:45.062 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:45.062 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:11:45.062 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:45.063 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:11:45.063 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:45.063 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:11:45.063 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:45.063 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:45.064 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:45.064 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:45.064 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:11:45.064 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:45.065 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:45.065 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:11:45.065 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:45.065 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:45.066 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:45.066 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:11:45.066 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:45.066 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:45.066 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:45.067 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:45.067 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:11:45.067 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:45.067 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:45.068 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:11:45.068 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:45.068 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:45.068 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:45.069 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:11:45.069 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:45.070 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:45.070 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:45.070 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:45.071 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:11:45.071 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:45.071 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:45.071 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:11:45.072 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:45.072 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:45.072 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:45.072 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:11:45.073 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:45.073 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:45.073 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:45.073 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:11:45.073 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:45.074 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:11:45.074 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:11:45.074 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:11:45.074 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:45.075 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:11:45.076 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:11:50.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:50.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:50.079 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:11:50.127 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:11:50.127 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:11:50.204 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:11:50.205 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:11:50.205 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:11:50.205 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:11:50.205 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:11:50.205 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:11:50.206 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:50.206 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:50.206 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:11:50.206 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:50.207 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:11:50.207 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:50.207 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:50.207 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:11:50.207 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:50.208 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:11:50.208 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:50.208 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:11:50.208 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:50.209 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:50.209 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:50.209 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:50.209 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:11:50.209 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:50.210 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:50.210 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:11:50.210 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:50.210 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:50.211 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:50.211 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:11:50.211 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:50.211 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:50.211 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:50.212 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:50.212 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:11:50.212 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:50.212 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:50.212 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.788603_210.1", 2021-09-16T20:11:50.213 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:50.213 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:50.213 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:50.213 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.788603Z", 2021-09-16T20:11:50.214 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:50.214 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:50.214 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:50.214 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:50.214 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:11:50.215 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:50.215 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:50.215 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:11:50.215 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:50.216 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:50.216 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:50.216 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:11:50.216 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:50.216 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:50.217 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:50.217 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:11:50.217 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:50.217 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:11:50.217 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:11:50.218 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:11:50.218 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:50.218 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:11:50.219 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.788603_210.1 2021-09-16T20:11:55.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:55.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:11:55.222 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:11:55.271 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:11:55.271 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:11:55.348 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:11:55.349 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:11:55.349 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:11:55.349 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:11:55.349 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:11:55.350 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:11:55.350 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:55.350 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:55.350 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:11:55.351 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:55.351 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:11:55.351 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:55.351 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:55.351 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:11:55.352 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:55.352 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:11:55.352 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:55.352 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:11:55.353 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:55.353 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:55.353 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:55.353 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:55.353 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:11:55.354 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:55.354 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:55.354 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:11:55.354 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:55.355 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:55.355 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:55.355 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:11:55.355 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:55.356 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:55.356 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:55.356 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:55.356 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:11:55.356 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:55.357 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:55.357 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.791325_211.1", 2021-09-16T20:11:55.357 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:55.357 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:55.357 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:55.358 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.791325Z", 2021-09-16T20:11:55.358 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:55.358 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:55.358 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:55.359 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:11:55.359 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:11:55.359 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:11:55.359 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:11:55.359 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:11:55.360 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:11:55.360 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:11:55.360 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:11:55.360 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:11:55.361 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:11:55.361 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:55.361 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:55.361 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:11:55.361 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:11:55.362 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:11:55.362 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:11:55.362 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:11:55.362 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:11:55.363 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:11:55.364 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.791325_211.1 2021-09-16T20:12:00.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:00.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:00.366 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:12:00.436 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:12:00.436 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:12:00.843 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:12:00.844 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:12:00.844 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:12:00.844 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:12:00.844 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:12:00.845 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:12:00.845 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:00.845 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:00.845 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:12:00.845 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:00.846 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:12:00.846 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:00.846 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:00.846 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:12:00.847 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:00.847 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:12:00.847 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:00.847 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:12:00.847 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:00.848 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:00.848 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:00.848 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:00.848 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:12:00.848 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:00.849 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:00.849 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:12:00.849 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:00.849 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:00.850 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:00.850 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:12:00.850 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:00.850 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:00.850 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:00.851 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:00.851 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:12:00.851 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:00.851 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:00.851 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:12:00.852 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:00.852 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:00.852 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:00.852 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:12:00.852 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:00.853 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:00.853 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:00.853 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:00.853 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:12:00.853 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:00.854 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:00.854 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:12:00.854 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:00.854 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:00.855 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:00.855 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:12:00.855 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:00.855 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:00.856 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:00.856 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:12:00.856 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:00.856 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:12:00.856 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:12:00.857 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:12:00.857 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:00.857 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:12:00.859 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:12:05.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:05.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:05.861 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:12:05.925 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:12:05.925 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:12:06.029 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:12:06.029 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:12:06.029 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:12:06.029 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:12:06.030 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:12:06.030 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:12:06.030 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:06.031 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:06.031 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:12:06.032 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:06.032 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:12:06.032 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:06.032 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:06.033 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:12:06.033 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:06.033 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:12:06.033 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:06.034 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:12:06.034 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:06.034 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:06.034 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:06.034 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:06.035 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:12:06.035 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:06.035 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:06.035 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:12:06.036 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:06.036 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:06.036 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:06.036 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:12:06.037 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:06.037 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:06.037 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:06.037 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:06.038 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:12:06.038 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:06.038 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:06.038 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.891592_214.1", 2021-09-16T20:12:06.039 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:06.039 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:06.039 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:06.039 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.891592Z", 2021-09-16T20:12:06.039 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:06.040 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:06.040 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:06.040 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:06.040 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:12:06.041 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:06.041 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:06.041 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:12:06.041 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:06.042 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:06.042 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:06.042 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:12:06.042 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:06.043 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:06.043 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:06.043 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:12:06.043 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:06.044 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:12:06.044 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:12:06.044 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:12:06.044 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:06.044 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:12:06.046 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.891592_214.1 2021-09-16T20:12:11.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:11.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:11.048 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:12:11.090 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:12:11.091 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:12:11.152 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:12:11.153 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:12:11.153 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:12:11.153 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:12:11.153 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:12:11.154 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:12:11.154 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:11.154 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:11.155 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:12:11.155 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:11.155 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:12:11.155 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:11.156 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:11.156 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:12:11.156 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:11.156 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:12:11.157 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:11.157 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:12:11.157 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:11.157 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:11.157 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:11.158 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:11.158 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:12:11.158 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:11.158 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:11.159 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:12:11.159 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:11.159 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:11.159 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:11.159 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:12:11.160 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:11.160 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:11.160 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:11.160 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:11.161 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:12:11.161 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:11.161 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:11.161 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:12:11.161 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:11.162 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:11.162 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:11.162 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:12:11.162 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:11.162 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:11.163 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:11.163 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:11.163 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:12:11.163 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:11.163 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:11.164 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:12:11.164 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:11.164 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:11.164 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:11.165 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:12:11.165 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:11.165 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:11.165 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:11.165 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:12:11.166 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:11.166 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:12:11.166 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:12:11.166 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:12:11.167 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:11.167 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:12:11.168 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:12:16.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:16.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:16.171 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:12:16.222 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:12:16.223 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:12:16.292 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:12:16.292 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:12:16.292 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:12:16.293 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:12:16.293 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:12:16.293 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:12:16.293 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:16.293 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:16.294 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:12:16.294 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:16.294 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:12:16.294 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:16.295 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:16.295 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:12:16.295 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:16.295 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:12:16.296 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:16.296 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:12:16.296 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:16.296 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:16.297 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:16.297 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:16.297 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:12:16.297 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:16.297 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:16.298 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:12:16.298 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:16.298 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:16.298 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:16.299 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:12:16.299 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:16.299 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:16.299 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:16.299 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:16.300 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:12:16.300 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:16.300 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:16.300 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.788603_210.1", 2021-09-16T20:12:16.301 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:16.301 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:16.301 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:16.301 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.788603Z", 2021-09-16T20:12:16.301 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:16.302 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:16.302 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:16.302 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:16.302 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:12:16.303 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:16.303 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:16.303 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:12:16.303 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:16.303 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:16.304 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:16.304 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:12:16.304 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:16.304 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:16.304 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:16.305 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:12:16.305 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:16.305 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:12:16.305 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:12:16.306 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:12:16.306 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:16.306 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:12:16.307 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.788603_210.1 2021-09-16T20:12:21.309 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... FAIL 2021-09-16T20:12:21.311 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:21.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:21.312 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:12:21.395 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:12:21.396 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:12:21.544 INFO:teuthology.orchestra.run.smithi109.stderr:No --period given, using current period=db0b3d90-59a0-4d0b-99ed-8408dd9cf54f 2021-09-16T20:12:21.582 INFO:teuthology.orchestra.run.smithi109.stdout:[ 2021-09-16T20:12:21.583 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:21.583 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:12:21.583 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:39:44.045455Z" 2021-09-16T20:12:21.584 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:21.584 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:21.584 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:12:21.584 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "0.000000" 2021-09-16T20:12:21.584 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:21.585 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:21.585 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631822766.419765_313.1", 2021-09-16T20:12:21.585 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T20:06:06.419765Z" 2021-09-16T20:12:21.585 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:21.586 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:21.586 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:12:21.586 INFO:teuthology.orchestra.run.smithi109.stdout: "last_update": "2021-09-16T19:42:17.633408Z" 2021-09-16T20:12:21.586 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:21.586 INFO:teuthology.orchestra.run.smithi109.stdout:] 2021-09-16T20:12:21.588 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-16T20:12:21.588 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:21.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:21.589 DEBUG:teuthology.orchestra.run.smithi118:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:12:21.642 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:12:21.643 INFO:teuthology.orchestra.run.smithi118.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:12:21.710 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2021-09-16T20:12:21.711 INFO:teuthology.orchestra.run.smithi118.stdout: "sync_status": { 2021-09-16T20:12:21.711 INFO:teuthology.orchestra.run.smithi118.stdout: "info": { 2021-09-16T20:12:21.711 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "sync", 2021-09-16T20:12:21.711 INFO:teuthology.orchestra.run.smithi118.stdout: "num_shards": 4, 2021-09-16T20:12:21.712 INFO:teuthology.orchestra.run.smithi118.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:12:21.712 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:12:21.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:12:21.713 INFO:teuthology.orchestra.run.smithi118.stdout: "markers": [ 2021-09-16T20:12:21.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:12:21.713 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 0, 2021-09-16T20:12:21.713 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:12:21.713 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:12:21.714 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:12:21.714 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:12:21.714 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 1, 2021-09-16T20:12:21.714 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:12:21.715 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:12:21.715 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:12:21.715 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:12:21.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:12:21.715 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:12:21.716 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 1, 2021-09-16T20:12:21.716 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:12:21.716 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:12:21.716 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "", 2021-09-16T20:12:21.717 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:12:21.717 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:12:21.717 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:12:21.717 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "0.000000", 2021-09-16T20:12:21.718 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:12:21.718 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:12:21.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:12:21.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:12:21.718 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 2, 2021-09-16T20:12:21.719 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:12:21.719 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:12:21.719 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631822766.419765_313.1", 2021-09-16T20:12:21.719 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:12:21.720 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:12:21.720 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:12:21.720 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T20:06:06.419765Z", 2021-09-16T20:12:21.720 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:12:21.720 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:12:21.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:12:21.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2021-09-16T20:12:21.721 INFO:teuthology.orchestra.run.smithi118.stdout: "key": 3, 2021-09-16T20:12:21.721 INFO:teuthology.orchestra.run.smithi118.stdout: "val": { 2021-09-16T20:12:21.722 INFO:teuthology.orchestra.run.smithi118.stdout: "state": 1, 2021-09-16T20:12:21.722 INFO:teuthology.orchestra.run.smithi118.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:12:21.722 INFO:teuthology.orchestra.run.smithi118.stdout: "next_step_marker": "", 2021-09-16T20:12:21.722 INFO:teuthology.orchestra.run.smithi118.stdout: "total_entries": 0, 2021-09-16T20:12:21.723 INFO:teuthology.orchestra.run.smithi118.stdout: "pos": 0, 2021-09-16T20:12:21.723 INFO:teuthology.orchestra.run.smithi118.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:12:21.723 INFO:teuthology.orchestra.run.smithi118.stdout: "realm_epoch": 2 2021-09-16T20:12:21.723 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:12:21.723 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:12:21.724 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2021-09-16T20:12:21.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2021-09-16T20:12:21.724 INFO:teuthology.orchestra.run.smithi118.stdout: "full_sync": { 2021-09-16T20:12:21.724 INFO:teuthology.orchestra.run.smithi118.stdout: "total": 1, 2021-09-16T20:12:21.725 INFO:teuthology.orchestra.run.smithi118.stdout: "complete": 1 2021-09-16T20:12:21.725 INFO:teuthology.orchestra.run.smithi118.stdout: } 2021-09-16T20:12:21.725 INFO:teuthology.orchestra.run.smithi118.stdout:} 2021-09-16T20:12:21.725 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-16T20:12:21.726 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-16T20:12:21.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:21.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:21.728 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:12:21.795 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:12:21.795 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:12:21.909 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:12:21.910 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:12:21.910 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:12:21.910 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:12:21.910 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:12:21.911 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:12:21.911 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:21.911 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:21.911 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:12:21.912 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:21.912 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:12:21.912 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:21.912 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:21.912 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:12:21.913 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:21.913 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:12:21.913 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:21.913 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:12:21.913 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:21.914 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:21.914 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:21.914 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:21.914 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:12:21.915 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:21.915 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:21.915 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:12:21.915 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:21.915 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:21.916 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:21.916 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:12:21.916 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:21.916 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:21.917 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:21.917 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:21.917 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:12:21.917 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:21.918 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:21.918 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.768089_209.1", 2021-09-16T20:12:21.918 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:21.918 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:21.918 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:21.919 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.768089Z", 2021-09-16T20:12:21.919 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:21.919 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:21.919 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:21.920 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:21.920 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:12:21.921 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:21.921 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:21.921 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:12:21.921 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:21.922 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:21.922 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:21.922 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:12:21.922 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:21.923 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:21.923 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:21.923 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:12:21.923 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:21.923 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:12:21.924 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:12:21.924 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:12:21.924 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:21.925 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:12:21.926 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.768089_209.1 2021-09-16T20:12:26.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:26.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:26.929 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:12:26.970 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:12:26.971 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:12:27.039 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:12:27.040 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:12:27.040 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:12:27.040 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:12:27.041 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:12:27.041 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:12:27.041 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:27.041 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:27.041 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:12:27.042 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:27.042 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:12:27.042 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:27.042 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:27.043 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:12:27.043 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:27.043 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:12:27.043 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:27.043 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:12:27.044 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:27.044 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:27.044 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:27.044 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:27.045 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:12:27.045 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:27.045 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:27.045 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:12:27.045 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:27.046 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:27.046 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:27.046 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:12:27.047 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:27.047 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:27.047 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:27.047 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:27.048 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:12:27.048 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:27.048 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:27.048 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.768089_209.1", 2021-09-16T20:12:27.048 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:27.049 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:27.049 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:27.049 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.768089Z", 2021-09-16T20:12:27.049 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:27.049 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:27.050 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:27.050 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:27.050 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:12:27.050 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:27.051 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:27.051 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:12:27.051 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:27.051 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:27.051 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:27.052 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:12:27.052 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:27.052 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:27.052 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:27.053 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:12:27.053 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:27.053 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:12:27.053 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:12:27.053 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:12:27.054 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:27.054 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:12:27.055 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.768089_209.1 2021-09-16T20:12:32.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:32.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:32.058 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:12:32.107 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:12:32.107 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:12:32.164 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:12:32.165 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:12:32.165 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:12:32.165 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:12:32.166 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:12:32.166 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:12:32.166 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:32.166 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:32.167 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:12:32.167 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:32.167 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:12:32.167 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:32.167 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:32.168 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:12:32.168 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:32.168 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:12:32.168 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:32.169 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:12:32.169 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:32.169 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:32.169 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:32.169 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:32.170 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:12:32.170 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:32.170 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:32.170 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:12:32.171 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:32.171 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:32.171 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:32.171 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:12:32.172 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:32.172 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:32.172 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:32.172 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:32.172 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:12:32.173 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:32.173 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:32.173 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.791325_211.1", 2021-09-16T20:12:32.173 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:32.173 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:32.174 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:32.174 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.791325Z", 2021-09-16T20:12:32.174 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:32.174 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:32.175 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:32.175 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:32.175 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:12:32.175 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:32.176 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:32.176 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:12:32.176 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:32.176 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:32.176 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:32.177 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:12:32.177 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:32.177 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:32.177 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:32.177 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:12:32.178 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:32.178 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:12:32.178 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:12:32.178 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:12:32.179 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:32.179 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:12:32.180 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.791325_211.1 2021-09-16T20:12:37.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:37.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:37.183 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:12:37.230 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:12:37.230 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:12:37.286 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:12:37.286 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:12:37.287 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:12:37.287 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:12:37.287 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:12:37.288 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:12:37.288 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:37.288 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:37.289 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:12:37.289 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:37.289 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:12:37.290 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:37.290 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:37.291 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:12:37.291 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:37.291 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:12:37.292 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:37.292 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:12:37.292 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:37.293 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:37.293 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:37.294 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:37.294 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:12:37.294 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:37.295 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:37.295 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:12:37.295 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:37.296 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:37.296 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:37.296 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:12:37.297 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:37.297 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:37.297 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:37.298 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:37.298 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:12:37.298 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:37.298 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:37.299 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.841894_213.1", 2021-09-16T20:12:37.299 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:37.299 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:37.299 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:37.300 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.841894Z", 2021-09-16T20:12:37.300 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:37.300 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:37.300 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:37.301 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:37.301 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:12:37.301 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:37.301 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:37.302 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:12:37.302 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:37.302 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:37.302 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:37.302 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:12:37.303 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:37.303 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:37.303 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:37.303 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:12:37.304 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:37.304 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:12:37.304 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:12:37.304 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:12:37.304 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:37.305 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:12:37.306 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.841894_213.1 2021-09-16T20:12:42.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:42.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:42.309 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:12:42.389 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:12:42.389 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:12:42.503 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:12:42.503 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:12:42.503 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:12:42.504 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:12:42.504 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:12:42.504 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:12:42.504 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:42.504 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:42.505 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:12:42.505 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:42.505 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:12:42.505 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:42.506 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:42.506 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:12:42.506 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:42.506 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:12:42.506 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:42.507 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:12:42.507 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:42.507 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:42.507 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:42.508 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:42.508 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:12:42.508 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:42.508 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:42.508 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:12:42.509 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:42.509 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:42.509 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:42.509 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:12:42.509 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:42.510 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:42.510 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:42.510 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:42.510 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:12:42.511 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:42.511 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:42.511 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.839033_212.1", 2021-09-16T20:12:42.511 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:42.512 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:42.512 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:42.512 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.839033Z", 2021-09-16T20:12:42.512 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:42.512 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:42.513 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:42.513 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:42.513 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:12:42.513 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:42.514 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:42.514 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:12:42.514 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:42.514 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:42.515 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:42.515 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:12:42.515 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:42.515 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:42.515 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:42.516 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:12:42.516 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:42.516 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:12:42.516 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:12:42.517 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:12:42.517 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:42.517 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:12:42.519 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.839033_212.1 2021-09-16T20:12:47.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:47.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:47.522 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:12:47.564 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:12:47.565 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:12:47.634 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:12:47.634 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:12:47.634 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:12:47.635 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:12:47.635 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:12:47.635 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:12:47.635 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:47.635 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:47.636 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:12:47.636 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:47.636 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:12:47.636 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:47.637 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:47.637 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:12:47.637 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:47.637 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:12:47.637 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:47.638 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:12:47.638 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:47.638 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:47.638 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:47.639 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:47.639 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:12:47.639 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:47.639 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:47.639 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:12:47.640 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:47.640 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:47.640 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:47.640 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:12:47.641 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:47.641 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:47.641 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:47.641 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:47.641 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:12:47.642 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:47.642 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:47.643 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:12:47.643 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:47.643 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:47.644 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:47.644 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:12:47.644 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:47.644 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:47.645 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:47.645 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:47.645 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:12:47.645 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:47.645 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:47.646 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:12:47.646 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:47.646 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:47.646 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:47.647 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:12:47.647 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:47.647 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:47.647 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:47.647 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:12:47.648 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:47.648 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:12:47.648 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:12:47.648 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:12:47.648 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:47.649 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:12:47.650 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:12:52.652 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:52.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:52.653 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:12:52.704 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:12:52.705 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:12:52.770 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:12:52.771 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:12:52.771 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:12:52.772 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:12:52.772 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:12:52.772 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:12:52.773 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:52.773 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:52.774 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:12:52.774 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:52.775 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:12:52.775 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:52.776 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:52.776 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:12:52.777 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:52.777 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:12:52.777 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:52.778 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:12:52.778 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:52.779 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:52.779 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:52.779 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:52.780 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:12:52.781 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:52.781 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:52.781 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:12:52.782 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:52.783 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:52.783 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:52.783 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:12:52.784 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:52.784 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:52.785 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:52.785 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:52.785 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:12:52.786 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:52.787 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:52.787 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.768089_209.1", 2021-09-16T20:12:52.787 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:52.787 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:52.788 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:52.788 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.768089Z", 2021-09-16T20:12:52.788 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:52.788 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:52.789 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:52.789 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:52.789 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:12:52.789 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:52.790 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:52.790 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:12:52.790 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:52.790 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:52.791 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:52.791 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:12:52.791 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:52.791 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:52.791 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:52.792 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:12:52.792 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:52.792 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:12:52.792 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:12:52.793 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:12:52.793 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:52.793 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:12:52.794 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.768089_209.1 2021-09-16T20:12:57.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:57.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:12:57.797 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:12:57.848 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:12:57.849 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:12:57.913 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:12:57.913 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:12:57.914 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:12:57.914 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:12:57.914 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:12:57.914 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:12:57.915 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:57.915 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:57.915 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:12:57.915 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:57.916 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:12:57.916 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:57.916 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:57.916 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:12:57.917 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:57.917 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:12:57.917 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:57.917 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:12:57.917 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:57.918 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:57.918 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:57.918 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:57.918 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:12:57.919 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:57.919 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:57.919 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:12:57.919 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:57.919 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:57.920 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:57.920 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:12:57.920 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:57.920 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:57.921 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:57.921 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:57.921 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:12:57.921 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:57.921 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:57.922 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.895465_215.1", 2021-09-16T20:12:57.922 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:57.922 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:57.922 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:57.922 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.895465Z", 2021-09-16T20:12:57.923 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:57.923 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:57.923 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:57.923 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:12:57.923 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:12:57.924 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:12:57.924 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:12:57.924 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:12:57.924 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:12:57.925 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:12:57.925 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:12:57.925 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:12:57.925 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:12:57.925 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:57.926 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:57.926 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:12:57.926 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:12:57.926 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:12:57.927 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:12:57.927 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:12:57.927 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:12:57.927 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:12:57.928 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.895465_215.1 2021-09-16T20:13:02.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:02.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:02.931 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:13:03.014 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:13:03.014 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:13:03.293 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:13:03.293 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:13:03.293 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:13:03.293 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:13:03.294 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:13:03.294 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:13:03.294 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:03.294 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:03.295 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:13:03.295 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:03.295 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:13:03.295 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:03.296 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:03.296 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:13:03.296 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:03.296 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:13:03.296 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:03.297 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:13:03.297 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:03.297 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:03.297 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:03.298 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:03.298 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:13:03.298 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:03.298 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:03.299 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:13:03.299 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:03.299 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:03.299 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:03.299 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:13:03.300 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:03.300 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:03.300 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:03.300 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:03.300 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:13:03.301 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:03.301 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:03.301 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821519.016744_220.1", 2021-09-16T20:13:03.301 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:03.301 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:03.302 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:03.302 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:19.016744Z", 2021-09-16T20:13:03.302 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:03.302 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:03.303 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:03.303 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:03.303 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:13:03.303 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:03.304 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:03.304 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:13:03.304 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:03.304 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:03.304 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:03.305 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:13:03.305 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:03.305 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:03.305 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:03.306 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:13:03.306 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:03.306 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:13:03.306 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:13:03.306 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:13:03.307 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:03.307 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:13:03.308 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821519.016744_220.1 2021-09-16T20:13:08.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:08.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:08.311 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:13:08.366 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:13:08.366 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:13:08.418 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:13:08.418 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:13:08.418 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:13:08.419 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:13:08.419 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:13:08.419 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:13:08.419 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:08.420 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:08.420 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:13:08.420 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:08.420 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:13:08.420 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:08.421 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:08.421 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:13:08.421 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:08.421 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:13:08.422 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:08.422 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:13:08.422 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:08.422 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:08.422 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:08.423 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:08.423 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:13:08.423 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:08.423 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:08.423 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:13:08.424 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:08.424 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:08.424 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:08.424 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:13:08.425 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:08.425 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:08.425 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:08.425 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:08.425 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:13:08.426 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:08.426 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:08.426 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:13:08.426 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:08.426 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:08.427 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:08.427 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:13:08.427 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:08.427 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:08.427 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:08.428 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:08.428 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:13:08.428 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:08.428 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:08.429 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:13:08.429 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:08.429 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:08.429 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:08.429 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:13:08.429 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:08.430 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:08.430 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:08.430 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:13:08.430 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:08.431 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:13:08.431 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:13:08.431 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:13:08.431 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:08.432 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:13:08.433 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:13:13.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:13.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:13.436 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:13:13.480 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:13:13.481 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:13:13.534 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:13:13.535 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:13:13.535 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:13:13.535 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:13:13.536 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:13:13.536 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:13:13.536 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:13.536 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:13.536 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:13:13.537 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:13.537 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:13:13.537 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:13.537 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:13.538 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:13:13.538 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:13.538 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:13:13.538 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:13.538 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:13:13.539 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:13.539 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:13.539 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:13.539 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:13.540 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:13:13.540 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:13.540 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:13.541 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:13:13.541 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:13.541 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:13.541 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:13.542 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:13:13.542 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:13.542 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:13.542 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:13.542 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:13.543 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:13:13.543 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:13.543 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:13.543 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821519.032233_221.1", 2021-09-16T20:13:13.544 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:13.544 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:13.544 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:13.545 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:19.032233Z", 2021-09-16T20:13:13.545 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:13.545 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:13.545 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:13.546 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:13.546 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:13:13.546 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:13.546 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:13.547 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:13:13.547 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:13.547 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:13.547 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:13.547 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:13:13.548 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:13.548 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:13.548 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:13.548 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:13:13.548 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:13.549 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:13:13.549 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:13:13.549 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:13:13.549 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:13.550 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:13:13.551 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821519.032233_221.1 2021-09-16T20:13:18.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:18.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:18.554 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:13:18.601 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:13:18.601 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:13:18.674 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:13:18.675 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:13:18.675 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:13:18.675 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:13:18.676 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:13:18.676 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:13:18.676 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:18.676 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:18.676 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:13:18.677 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:18.677 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:13:18.677 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:18.677 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:18.678 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:13:18.678 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:18.678 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:13:18.678 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:18.678 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:13:18.679 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:18.679 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:18.679 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:18.679 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:18.680 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:13:18.680 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:18.680 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:18.680 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:13:18.681 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:18.681 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:18.681 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:18.681 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:13:18.681 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:18.682 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:18.682 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:18.682 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:18.682 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:13:18.683 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:18.683 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:18.683 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.957515_216.1", 2021-09-16T20:13:18.683 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:18.684 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:18.684 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:18.684 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.957515Z", 2021-09-16T20:13:18.684 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:18.684 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:18.685 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:18.685 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:18.685 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:13:18.685 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:18.686 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:18.686 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:13:18.686 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:18.686 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:18.686 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:18.687 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:13:18.687 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:18.687 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:18.687 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:18.688 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:13:18.688 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:18.688 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:13:18.688 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:13:18.688 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:13:18.689 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:18.689 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:13:18.690 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.957515_216.1 2021-09-16T20:13:23.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:23.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:23.693 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:13:23.787 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:13:23.788 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:13:24.175 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:13:24.175 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:13:24.176 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:13:24.176 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:13:24.176 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:13:24.176 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:13:24.177 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:24.177 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:24.177 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:13:24.177 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:24.177 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:13:24.178 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:24.178 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:24.178 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:13:24.178 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:24.179 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:13:24.179 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:24.179 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:13:24.179 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:24.180 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:24.180 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:24.180 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:24.180 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:13:24.180 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:24.181 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:24.181 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:13:24.181 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:24.181 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:24.182 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:24.182 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:13:24.182 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:24.182 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:24.182 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:24.183 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:24.183 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:13:24.183 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:24.183 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:24.184 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.895465_215.1", 2021-09-16T20:13:24.184 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:24.184 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:24.184 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:24.185 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.895465Z", 2021-09-16T20:13:24.185 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:24.185 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:24.185 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:24.185 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:24.186 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:13:24.186 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:24.186 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:24.186 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:13:24.187 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:24.187 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:24.187 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:24.187 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:13:24.187 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:24.188 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:24.188 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:24.188 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:13:24.188 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:24.189 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:13:24.189 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:13:24.189 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:13:24.189 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:24.190 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:13:24.191 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.895465_215.1 2021-09-16T20:13:29.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:29.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:29.194 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:13:29.243 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:13:29.243 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:13:29.306 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:13:29.307 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:13:29.307 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:13:29.307 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:13:29.307 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:13:29.308 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:13:29.308 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:29.308 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:29.308 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:13:29.309 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:29.309 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:13:29.309 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:29.309 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:29.310 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:13:29.310 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:29.310 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:13:29.310 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:29.311 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:13:29.311 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:29.311 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:29.311 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:29.312 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:29.312 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:13:29.312 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:29.312 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:29.312 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:13:29.313 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:29.313 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:29.313 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:29.313 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:13:29.314 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:29.314 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:29.314 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:29.314 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:29.314 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:13:29.315 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:29.315 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:29.315 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.768089_209.1", 2021-09-16T20:13:29.315 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:29.316 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:29.316 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:29.316 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.768089Z", 2021-09-16T20:13:29.316 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:29.316 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:29.317 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:29.317 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:29.317 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:13:29.317 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:29.318 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:29.318 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:13:29.318 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:29.318 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:29.318 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:29.319 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:13:29.319 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:29.319 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:29.319 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:29.319 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:13:29.320 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:29.320 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:13:29.320 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:13:29.320 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:13:29.321 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:29.321 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:13:29.322 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.768089_209.1 2021-09-16T20:13:34.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:34.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:34.325 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:13:34.371 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:13:34.371 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:13:34.442 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:13:34.442 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:13:34.443 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:13:34.443 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:13:34.443 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:13:34.443 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:13:34.444 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:34.444 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:34.444 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:13:34.444 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:34.445 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:13:34.445 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:34.445 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:34.445 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:13:34.446 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:34.446 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:13:34.446 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:34.446 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:13:34.446 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:34.447 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:34.447 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:34.447 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:34.447 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:13:34.448 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:34.448 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:34.448 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:13:34.448 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:34.448 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:34.449 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:34.449 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:13:34.449 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:34.450 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:34.450 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:34.450 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:34.450 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:13:34.451 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:34.451 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:34.451 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.957515_216.1", 2021-09-16T20:13:34.451 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:34.452 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:34.452 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:34.452 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.957515Z", 2021-09-16T20:13:34.452 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:34.452 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:34.453 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:34.453 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:34.453 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:13:34.453 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:34.454 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:34.454 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:13:34.454 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:34.454 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:34.454 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:34.455 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:13:34.455 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:34.455 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:34.455 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:34.456 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:13:34.456 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:34.456 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:13:34.456 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:13:34.456 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:13:34.457 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:34.457 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:13:34.458 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.957515_216.1 2021-09-16T20:13:39.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:39.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:39.461 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:13:39.526 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:13:39.527 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:13:39.882 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:13:39.882 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:13:39.882 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:13:39.883 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:13:39.883 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:13:39.883 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:13:39.883 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:39.883 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:39.884 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:13:39.884 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:39.884 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:13:39.884 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:39.884 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:39.885 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:13:39.885 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:39.885 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:13:39.885 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:39.886 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:13:39.886 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:39.886 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:39.886 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:39.886 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:39.887 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:13:39.887 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:39.887 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:39.887 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:13:39.887 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:39.888 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:39.888 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:39.888 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:13:39.888 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:39.888 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:39.889 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:39.889 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:39.889 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:13:39.889 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:39.890 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:39.890 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:13:39.890 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:39.890 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:39.890 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:39.891 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:13:39.891 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:39.891 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:39.891 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:39.891 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:39.892 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:13:39.892 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:39.892 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:39.892 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:13:39.892 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:39.893 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:39.893 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:39.893 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:13:39.893 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:39.894 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:39.894 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:39.894 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:13:39.894 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:39.894 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:13:39.895 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:13:39.895 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:13:39.895 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:39.895 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:13:39.896 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:13:44.898 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:44.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:44.899 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:13:44.964 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:13:44.964 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:13:45.072 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:13:45.073 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:13:45.073 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:13:45.073 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:13:45.074 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:13:45.074 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:13:45.074 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:45.074 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:45.074 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:13:45.075 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:45.075 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:13:45.075 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:45.075 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:45.076 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:13:45.076 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:45.076 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:13:45.076 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:45.076 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:13:45.077 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:45.077 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:45.077 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:45.077 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:45.077 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:13:45.078 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:45.078 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:45.078 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:13:45.078 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:45.079 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:45.079 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:45.079 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:13:45.079 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:45.079 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:45.080 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:45.080 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:45.080 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:13:45.080 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:45.080 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:45.081 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.788603_210.1", 2021-09-16T20:13:45.081 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:45.081 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:45.081 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:45.082 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.788603Z", 2021-09-16T20:13:45.082 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:45.082 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:45.082 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:45.082 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:45.083 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:13:45.083 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:45.083 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:45.083 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:13:45.084 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:45.084 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:45.084 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:45.084 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:13:45.084 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:45.085 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:45.085 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:45.085 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:13:45.085 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:45.085 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:13:45.086 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:13:45.086 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:13:45.086 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:45.086 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:13:45.088 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.788603_210.1 2021-09-16T20:13:50.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:50.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:50.090 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:13:50.132 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:13:50.133 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:13:50.198 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:13:50.198 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:13:50.198 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:13:50.198 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:13:50.199 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:13:50.199 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:13:50.199 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:50.199 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:50.200 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:13:50.200 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:50.200 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:13:50.200 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:50.201 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:50.201 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:13:50.201 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:50.201 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:13:50.201 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:50.202 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:13:50.202 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:50.202 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:50.202 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:50.203 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:50.203 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:13:50.203 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:50.203 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:50.203 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:13:50.204 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:50.204 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:50.204 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:50.204 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:13:50.204 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:50.205 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:50.205 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:50.205 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:50.206 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:13:50.206 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:50.206 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:50.206 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821519.035503_222.1", 2021-09-16T20:13:50.207 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:50.207 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:50.207 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:50.207 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:19.035503Z", 2021-09-16T20:13:50.208 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:50.208 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:50.208 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:50.208 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:50.208 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:13:50.209 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:50.209 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:50.209 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:13:50.209 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:50.209 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:50.210 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:50.210 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:13:50.210 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:50.210 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:50.211 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:50.211 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:13:50.211 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:50.211 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:13:50.211 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:13:50.212 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:13:50.212 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:50.212 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:13:50.213 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821519.035503_222.1 2021-09-16T20:13:55.215 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:55.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:13:55.216 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:13:55.268 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:13:55.269 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:13:55.357 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:13:55.358 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:13:55.358 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:13:55.358 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:13:55.358 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:13:55.359 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:13:55.359 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:55.359 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:55.359 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:13:55.360 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:55.360 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:13:55.360 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:55.360 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:55.360 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:13:55.361 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:55.361 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:13:55.361 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:55.362 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:13:55.362 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:55.362 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:55.363 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:55.363 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:55.363 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:13:55.363 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:55.364 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:55.364 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:13:55.364 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:55.364 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:55.365 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:55.365 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:13:55.365 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:55.365 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:55.365 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:55.366 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:55.366 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:13:55.366 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:55.366 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:55.367 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.768089_209.1", 2021-09-16T20:13:55.367 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:55.367 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:55.367 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:55.367 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.768089Z", 2021-09-16T20:13:55.368 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:55.368 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:55.368 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:55.368 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:13:55.368 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:13:55.369 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:13:55.369 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:13:55.369 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:13:55.369 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:13:55.370 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:13:55.370 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:13:55.370 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:13:55.370 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:13:55.370 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:55.371 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:55.371 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:13:55.371 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:13:55.371 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:13:55.371 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:13:55.372 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:13:55.372 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:13:55.372 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:13:55.373 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.768089_209.1 2021-09-16T20:14:00.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:00.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:00.376 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:14:00.453 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:14:00.453 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:14:00.570 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:14:00.571 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:14:00.571 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:14:00.571 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:14:00.571 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:14:00.571 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:14:00.572 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:00.572 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:00.572 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:14:00.572 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:00.573 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:14:00.573 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:00.573 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:00.573 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:14:00.574 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:00.574 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:14:00.574 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:00.574 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:14:00.574 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:00.575 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:00.575 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:00.575 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:00.575 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:14:00.576 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:00.576 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:00.576 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:14:00.576 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:00.576 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:00.577 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:00.577 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:14:00.577 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:00.577 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:00.578 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:00.578 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:00.578 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:14:00.578 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:00.578 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:00.579 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:14:00.579 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:00.579 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:00.579 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:00.580 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:14:00.580 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:00.580 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:00.580 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:00.581 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:00.581 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:14:00.581 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:00.581 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:00.581 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:14:00.582 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:00.582 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:00.582 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:00.582 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:14:00.583 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:00.583 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:00.583 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:00.583 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:14:00.584 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:00.584 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:14:00.584 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:14:00.584 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:14:00.584 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:00.585 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:14:00.586 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:14:05.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:05.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:05.594 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:14:05.636 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:14:05.637 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:14:05.708 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:14:05.708 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:14:05.708 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:14:05.709 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:14:05.709 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:14:05.709 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:14:05.709 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:05.709 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:05.710 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:14:05.710 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:05.710 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:14:05.710 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:05.711 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:05.711 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:14:05.711 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:05.711 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:14:05.711 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:05.712 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:14:05.712 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:05.712 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:05.712 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:05.712 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:05.713 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:14:05.713 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:05.713 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:05.713 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:14:05.714 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:05.714 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:05.714 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:05.714 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:14:05.715 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:05.715 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:05.715 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:05.715 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:05.715 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:14:05.716 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:05.716 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:05.716 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:14:05.716 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:05.717 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:05.717 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:05.717 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:14:05.717 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:05.717 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:05.718 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:05.718 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:05.718 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:14:05.719 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:05.719 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:05.719 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:14:05.719 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:05.720 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:05.720 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:05.720 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:14:05.720 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:05.720 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:05.721 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:05.721 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:14:05.721 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:05.721 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:14:05.721 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:14:05.722 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:14:05.722 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:05.722 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:14:05.723 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:14:10.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:10.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:10.726 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:14:10.770 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:14:10.771 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:14:10.838 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:14:10.838 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:14:10.838 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:14:10.839 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:14:10.839 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:14:10.839 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:14:10.839 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:10.839 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:10.840 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:14:10.840 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:10.840 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:14:10.840 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:10.841 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:10.841 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:14:10.841 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:10.841 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:14:10.842 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:10.842 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:14:10.842 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:10.842 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:10.843 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:10.843 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:10.843 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:14:10.843 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:10.843 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:10.844 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:14:10.844 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:10.844 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:10.844 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:10.845 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:14:10.845 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:10.845 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:10.845 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:10.845 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:10.846 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:14:10.846 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:10.846 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:10.846 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:14:10.847 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:10.847 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:10.847 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:10.847 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:14:10.848 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:10.848 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:10.848 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:10.848 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:10.848 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:14:10.849 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:10.849 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:10.849 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:14:10.849 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:10.849 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:10.850 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:10.850 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:14:10.850 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:10.850 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:10.851 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:10.851 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:14:10.851 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:10.851 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:14:10.851 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:14:10.852 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:14:10.852 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:10.852 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:14:10.853 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:14:15.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:15.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:15.857 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:14:15.905 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:14:15.905 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:14:15.971 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:14:15.972 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:14:15.972 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:14:15.972 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:14:15.972 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:14:15.973 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:14:15.973 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:15.973 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:15.973 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:14:15.973 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:15.974 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:14:15.974 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:15.974 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:15.974 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:14:15.975 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:15.975 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:14:15.975 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:15.975 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:14:15.976 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:15.976 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:15.976 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:15.976 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:15.976 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:14:15.977 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:15.977 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:15.977 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:14:15.977 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:15.977 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:15.977 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:15.978 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:14:15.978 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:15.978 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:15.978 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:15.979 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:15.979 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:14:15.979 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:15.979 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:15.979 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:14:15.980 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:15.980 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:15.980 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:15.980 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:14:15.981 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:15.981 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:15.981 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:15.981 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:15.981 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:14:15.982 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:15.982 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:15.982 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:14:15.982 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:15.982 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:15.983 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:15.983 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:14:15.983 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:15.983 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:15.984 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:15.984 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:14:15.984 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:15.984 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:14:15.984 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:14:15.985 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:14:15.985 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:15.985 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:14:15.986 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:14:20.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:20.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:20.989 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:14:21.073 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:14:21.074 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:14:21.209 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:14:21.209 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:14:21.210 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:14:21.210 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:14:21.210 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:14:21.210 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:14:21.211 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:21.211 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:21.211 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:14:21.211 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:21.212 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:14:21.212 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:21.212 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:21.212 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:14:21.213 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:21.213 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:14:21.213 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:21.213 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:14:21.213 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:21.214 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:21.214 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:21.214 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:21.214 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:14:21.215 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:21.215 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:21.215 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:14:21.215 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:21.216 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:21.216 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:21.216 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:14:21.216 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:21.216 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:21.217 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:21.217 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:21.217 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:14:21.217 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:21.218 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:21.218 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.788603_210.1", 2021-09-16T20:14:21.218 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:21.218 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:21.219 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:21.219 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.788603Z", 2021-09-16T20:14:21.219 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:21.219 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:21.219 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:21.220 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:21.220 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:14:21.220 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:21.220 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:21.221 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:14:21.221 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:21.221 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:21.221 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:21.221 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:14:21.222 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:21.222 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:21.222 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:21.222 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:14:21.223 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:21.223 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:14:21.223 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:14:21.223 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:14:21.223 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:21.224 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:14:21.225 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.788603_210.1 2021-09-16T20:14:26.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:26.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:26.227 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:14:26.277 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:14:26.277 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:14:26.333 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:14:26.333 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:14:26.333 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:14:26.334 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:14:26.334 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:14:26.334 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:14:26.334 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:26.335 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:26.335 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:14:26.335 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:26.335 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:14:26.335 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:26.336 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:26.336 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:14:26.336 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:26.336 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:14:26.337 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:26.337 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:14:26.337 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:26.337 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:26.337 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:26.338 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:26.338 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:14:26.338 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:26.338 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:26.339 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:14:26.339 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:26.339 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:26.339 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:26.340 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:14:26.340 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:26.340 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:26.340 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:26.341 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:26.341 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:14:26.342 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:26.342 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:26.342 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:14:26.342 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:26.343 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:26.343 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:26.343 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:14:26.343 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:26.344 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:26.344 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:26.344 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:26.344 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:14:26.345 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:26.345 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:26.345 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:14:26.345 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:26.345 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:26.346 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:26.346 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:14:26.346 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:26.346 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:26.347 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:26.347 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:14:26.348 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:26.348 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:14:26.348 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:14:26.348 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:14:26.348 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:26.349 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:14:26.350 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:14:31.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:31.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:31.353 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:14:31.395 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:14:31.395 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:14:31.449 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:14:31.450 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:14:31.450 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:14:31.450 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:14:31.450 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:14:31.451 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:14:31.451 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:31.451 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:31.451 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:14:31.451 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:31.452 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:14:31.452 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:31.452 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:31.452 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:14:31.453 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:31.453 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:14:31.453 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:31.453 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:14:31.453 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:31.454 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:31.454 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:31.454 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:31.454 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:14:31.455 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:31.455 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:31.455 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:14:31.455 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:31.456 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:31.456 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:31.456 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:14:31.456 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:31.456 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:31.457 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:31.457 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:31.457 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:14:31.457 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:31.458 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:31.458 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.768089_209.1", 2021-09-16T20:14:31.458 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:31.458 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:31.458 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:31.459 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.768089Z", 2021-09-16T20:14:31.459 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:31.459 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:31.459 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:31.460 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:31.460 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:14:31.460 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:31.460 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:31.460 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:14:31.461 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:31.461 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:31.461 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:31.461 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:14:31.461 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:31.462 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:31.462 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:31.462 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:14:31.462 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:31.463 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:14:31.463 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:14:31.463 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:14:31.463 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:31.464 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:14:31.465 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.768089_209.1 2021-09-16T20:14:36.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:36.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:36.468 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:14:36.517 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:14:36.517 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:14:36.579 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:14:36.579 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:14:36.580 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:14:36.580 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:14:36.580 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:14:36.580 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:14:36.581 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:36.581 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:36.581 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:14:36.581 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:36.582 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:14:36.582 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:36.582 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:36.582 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:14:36.582 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:36.583 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:14:36.583 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:36.583 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:14:36.583 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:36.584 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:36.584 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:36.584 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:36.584 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:14:36.585 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:36.585 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:36.585 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:14:36.585 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:36.585 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:36.586 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:36.586 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:14:36.586 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:36.586 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:36.587 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:36.587 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:36.587 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:14:36.587 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:36.588 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:36.588 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.788603_210.1", 2021-09-16T20:14:36.588 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:36.588 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:36.588 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:36.589 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.788603Z", 2021-09-16T20:14:36.589 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:36.589 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:36.589 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:36.590 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:36.590 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:14:36.590 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:36.590 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:36.590 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:14:36.591 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:36.591 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:36.591 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:36.591 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:14:36.592 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:36.592 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:36.592 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:36.592 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:14:36.592 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:36.593 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:14:36.593 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:14:36.593 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:14:36.593 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:36.594 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:14:36.595 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.788603_210.1 2021-09-16T20:14:41.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:41.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:41.598 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:14:41.676 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:14:41.676 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:14:41.939 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:14:41.939 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:14:41.939 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:14:41.940 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:14:41.940 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:14:41.940 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:14:41.940 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:41.941 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:41.941 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:14:41.941 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:41.941 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:14:41.941 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:41.942 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:41.942 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:14:41.942 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:41.942 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:14:41.943 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:41.943 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:14:41.943 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:41.943 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:41.944 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:41.944 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:41.944 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:14:41.944 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:41.944 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:41.945 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:14:41.945 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:41.945 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:41.945 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:41.945 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:14:41.946 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:41.946 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:41.946 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:41.946 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:41.947 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:14:41.947 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:41.947 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:41.947 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.841894_213.1", 2021-09-16T20:14:41.947 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:41.948 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:41.948 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:41.948 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.841894Z", 2021-09-16T20:14:41.948 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:41.948 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:41.949 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:41.949 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:41.949 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:14:41.949 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:41.950 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:41.950 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:14:41.950 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:41.950 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:41.951 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:41.951 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:14:41.951 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:41.951 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:41.951 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:41.952 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:14:41.952 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:41.952 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:14:41.952 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:14:41.952 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:14:41.953 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:41.953 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:14:41.954 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.841894_213.1 2021-09-16T20:14:46.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:46.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:46.961 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:14:47.003 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:14:47.004 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:14:47.073 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:14:47.073 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:14:47.073 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:14:47.074 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:14:47.074 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:14:47.074 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:14:47.075 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:47.075 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:47.075 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:14:47.075 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:47.075 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:14:47.076 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:47.076 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:47.076 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:14:47.076 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:47.077 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:14:47.077 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:47.077 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:14:47.077 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:47.078 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:47.078 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:47.078 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:47.078 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:14:47.079 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:47.079 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:47.079 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:14:47.079 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:47.080 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:47.080 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:47.080 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:14:47.080 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:47.080 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:47.081 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:47.081 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:47.081 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:14:47.082 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:47.082 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:47.082 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.841894_213.1", 2021-09-16T20:14:47.082 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:47.082 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:47.083 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:47.083 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.841894Z", 2021-09-16T20:14:47.083 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:47.083 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:47.084 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:47.084 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:47.084 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:14:47.084 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:47.085 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:47.085 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:14:47.085 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:47.085 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:47.085 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:47.086 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:14:47.086 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:47.086 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:47.086 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:47.087 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:14:47.087 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:47.087 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:14:47.087 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:14:47.088 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:14:47.088 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:47.088 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:14:47.089 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.841894_213.1 2021-09-16T20:14:52.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:52.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:52.092 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:14:52.134 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:14:52.134 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:14:52.188 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:14:52.188 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:14:52.189 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:14:52.189 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:14:52.189 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:14:52.189 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:14:52.189 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:52.190 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:52.190 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:14:52.190 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:52.190 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:14:52.191 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:52.191 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:52.191 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:14:52.191 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:52.192 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:14:52.192 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:52.192 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:14:52.192 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:52.192 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:52.193 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:52.193 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:52.193 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:14:52.193 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:52.193 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:52.194 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:14:52.194 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:52.194 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:52.194 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:52.195 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:14:52.195 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:52.195 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:52.195 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:52.195 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:52.196 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:14:52.196 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:52.196 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:52.196 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.791325_211.1", 2021-09-16T20:14:52.197 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:52.197 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:52.197 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:52.197 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.791325Z", 2021-09-16T20:14:52.197 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:52.198 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:52.198 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:52.198 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:52.198 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:14:52.199 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:52.199 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:52.199 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:14:52.199 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:52.199 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:52.200 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:52.200 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:14:52.200 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:52.200 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:52.200 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:52.201 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:14:52.201 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:52.201 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:14:52.201 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:14:52.202 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:14:52.202 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:52.202 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:14:52.204 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.791325_211.1 2021-09-16T20:14:57.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:57.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:14:57.207 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:14:57.252 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:14:57.253 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:14:57.308 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:14:57.308 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:14:57.308 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:14:57.309 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:14:57.309 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:14:57.309 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:14:57.309 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:57.310 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:57.310 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:14:57.310 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:57.310 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:14:57.311 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:57.311 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:57.311 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:14:57.311 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:57.311 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:14:57.312 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:57.312 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:14:57.312 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:57.312 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:57.313 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:57.313 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:57.313 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:14:57.313 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:57.313 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:57.314 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:14:57.314 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:57.314 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:57.314 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:57.314 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:14:57.315 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:57.315 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:57.315 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:57.315 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:57.316 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:14:57.316 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:57.317 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:57.317 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.895465_215.1", 2021-09-16T20:14:57.317 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:57.318 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:57.318 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:57.319 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.895465Z", 2021-09-16T20:14:57.319 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:57.319 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:57.320 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:57.320 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:14:57.321 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:14:57.321 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:14:57.321 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:14:57.322 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:14:57.322 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:14:57.323 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:14:57.323 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:14:57.323 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:14:57.324 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:14:57.324 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:57.325 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:57.325 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:14:57.325 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:14:57.326 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:14:57.326 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:14:57.327 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:14:57.327 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:14:57.327 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:14:57.329 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.895465_215.1 2021-09-16T20:15:02.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:02.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:02.331 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:15:02.386 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:15:02.386 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:15:02.589 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:15:02.589 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:15:02.590 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:15:02.590 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:15:02.590 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:15:02.590 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:15:02.591 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:02.591 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:02.591 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:15:02.591 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:02.591 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:15:02.592 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:02.592 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:02.592 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:15:02.592 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:02.592 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:15:02.593 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:02.593 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:15:02.593 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:02.593 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:02.594 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:02.594 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:02.594 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:15:02.594 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:02.594 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:02.595 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:15:02.595 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:02.595 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:02.595 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:02.595 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:15:02.596 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:02.596 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:02.596 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:02.596 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:02.596 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:15:02.597 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:02.597 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:02.597 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821519.173860_229.1", 2021-09-16T20:15:02.597 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:02.598 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:02.598 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:02.598 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:19.173860Z", 2021-09-16T20:15:02.598 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:02.598 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:02.599 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:02.599 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:02.599 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:15:02.599 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:02.599 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:02.600 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:15:02.600 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:02.600 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:02.600 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:02.600 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:15:02.601 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:02.601 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:02.601 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:02.601 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:15:02.601 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:02.602 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:15:02.602 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:15:02.602 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:15:02.602 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:02.602 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:15:02.604 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821519.173860_229.1 2021-09-16T20:15:07.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:07.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:07.607 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:15:07.666 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:15:07.667 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:15:07.760 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:15:07.760 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:15:07.760 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:15:07.761 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:15:07.761 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:15:07.761 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:15:07.761 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:07.762 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:07.762 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:15:07.762 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:07.762 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:15:07.762 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:07.763 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:07.763 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:15:07.763 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:07.763 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:15:07.764 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:07.764 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:15:07.764 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:07.764 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:07.765 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:07.765 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:07.765 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:15:07.765 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:07.765 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:07.766 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:15:07.766 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:07.766 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:07.766 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:07.766 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:15:07.767 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:07.767 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:07.767 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:07.767 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:07.768 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:15:07.768 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:07.768 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:07.768 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.891592_214.1", 2021-09-16T20:15:07.768 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:07.769 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:07.769 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:07.769 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.891592Z", 2021-09-16T20:15:07.769 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:07.770 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:07.770 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:07.770 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:07.770 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:15:07.771 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:07.771 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:07.771 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:15:07.771 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:07.771 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:07.772 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:07.772 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:15:07.772 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:07.772 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:07.773 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:07.773 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:15:07.773 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:07.773 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:15:07.773 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:15:07.774 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:15:07.774 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:07.774 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:15:07.775 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.891592_214.1 2021-09-16T20:15:12.776 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:12.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:12.777 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:15:12.821 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:15:12.821 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:15:12.878 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:15:12.878 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:15:12.879 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:15:12.879 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:15:12.879 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:15:12.879 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:15:12.880 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:12.880 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:12.880 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:15:12.880 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:12.880 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:15:12.881 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:12.881 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:12.881 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:15:12.881 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:12.882 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:15:12.882 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:12.882 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:15:12.882 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:12.883 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:12.883 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:12.883 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:12.883 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:15:12.883 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:12.884 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:12.884 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:15:12.884 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:12.884 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:12.885 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:12.885 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:15:12.885 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:12.885 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:12.885 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:12.886 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:12.886 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:15:12.886 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:12.886 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:12.887 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.791325_211.1", 2021-09-16T20:15:12.887 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:12.887 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:12.887 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:12.887 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.791325Z", 2021-09-16T20:15:12.888 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:12.888 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:12.888 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:12.888 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:12.889 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:15:12.889 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:12.889 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:12.890 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:15:12.890 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:12.890 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:12.891 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:12.891 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:15:12.891 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:12.891 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:12.892 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:12.892 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:15:12.892 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:12.892 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:15:12.893 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:15:12.893 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:15:12.893 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:12.893 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:15:12.894 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.791325_211.1 2021-09-16T20:15:17.896 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:17.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:17.897 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:15:17.944 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:15:17.944 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:15:18.005 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:15:18.006 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:15:18.006 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:15:18.006 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:15:18.006 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:15:18.007 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:15:18.007 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:18.007 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:18.007 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:15:18.008 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:18.008 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:15:18.008 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:18.008 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:18.008 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:15:18.009 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:18.009 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:15:18.009 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:18.010 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:15:18.010 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:18.010 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:18.010 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:18.010 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:18.011 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:15:18.011 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:18.011 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:18.011 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:15:18.012 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:18.012 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:18.012 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:18.012 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:15:18.013 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:18.013 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:18.013 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:18.013 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:18.013 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:15:18.014 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:18.014 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:18.014 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821519.116095_226.1", 2021-09-16T20:15:18.015 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:18.015 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:18.015 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:18.015 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:19.116095Z", 2021-09-16T20:15:18.015 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:18.016 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:18.016 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:18.016 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:18.016 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:15:18.016 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:18.017 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:18.017 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:15:18.017 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:18.017 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:18.018 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:18.018 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:15:18.018 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:18.018 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:18.019 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:18.019 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:15:18.019 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:18.019 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:15:18.019 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:15:18.020 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:15:18.020 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:18.020 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:15:18.021 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821519.116095_226.1 2021-09-16T20:15:23.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:23.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:23.024 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:15:23.106 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:15:23.107 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:15:23.288 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:15:23.288 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:15:23.289 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:15:23.289 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:15:23.289 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:15:23.289 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:15:23.290 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:23.290 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:23.290 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:15:23.290 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:23.291 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:15:23.291 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:23.291 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:23.291 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:15:23.291 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:23.292 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:15:23.292 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:23.292 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:15:23.292 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:23.293 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:23.293 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:23.293 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:23.293 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:15:23.294 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:23.294 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:23.294 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:15:23.294 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:23.295 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:23.295 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:23.295 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:15:23.295 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:23.296 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:23.296 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:23.296 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:23.296 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:15:23.296 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:23.297 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:23.297 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.791325_211.1", 2021-09-16T20:15:23.297 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:23.297 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:23.297 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:23.298 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.791325Z", 2021-09-16T20:15:23.298 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:23.298 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:23.298 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:23.299 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:23.299 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:15:23.299 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:23.299 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:23.299 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:15:23.300 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:23.300 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:23.300 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:23.300 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:15:23.301 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:23.301 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:23.301 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:23.301 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:15:23.301 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:23.302 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:15:23.302 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:15:23.302 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:15:23.302 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:23.303 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:15:23.304 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.791325_211.1 2021-09-16T20:15:28.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:28.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:28.306 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:15:28.359 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:15:28.359 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:15:28.428 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:15:28.429 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:15:28.429 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:15:28.429 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:15:28.429 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:15:28.430 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:15:28.430 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:28.430 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:28.430 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:15:28.431 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:28.431 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:15:28.431 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:28.431 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:28.432 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:15:28.432 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:28.432 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:15:28.432 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:28.433 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:15:28.433 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:28.433 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:28.433 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:28.433 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:28.434 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:15:28.434 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:28.434 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:28.434 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:15:28.435 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:28.435 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:28.435 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:28.435 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:15:28.435 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:28.436 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:28.436 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:28.436 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:28.436 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:15:28.436 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:28.437 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:28.437 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.839033_212.1", 2021-09-16T20:15:28.437 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:28.437 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:28.438 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:28.438 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.839033Z", 2021-09-16T20:15:28.438 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:28.438 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:28.439 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:28.439 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:28.439 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:15:28.439 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:28.439 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:28.440 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:15:28.440 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:28.440 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:28.440 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:28.441 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:15:28.441 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:28.441 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:28.441 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:28.442 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:15:28.442 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:28.442 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:15:28.442 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:15:28.442 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:15:28.443 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:28.443 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:15:28.444 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.839033_212.1 2021-09-16T20:15:33.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:33.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:33.447 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:15:33.496 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:15:33.496 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:15:33.557 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:15:33.558 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:15:33.558 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:15:33.558 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:15:33.559 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:15:33.559 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:15:33.559 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:33.559 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:33.560 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:15:33.560 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:33.560 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:15:33.560 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:33.561 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:33.561 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:15:33.561 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:33.561 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:15:33.561 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:33.562 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:15:33.562 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:33.562 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:33.562 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:33.563 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:33.563 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:15:33.563 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:33.563 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:33.564 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:15:33.564 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:33.564 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:33.564 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:33.565 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:15:33.565 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:33.565 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:33.565 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:33.565 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:33.566 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:15:33.566 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:33.566 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:33.566 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.839033_212.1", 2021-09-16T20:15:33.567 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:33.567 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:33.567 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:33.567 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.839033Z", 2021-09-16T20:15:33.567 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:33.568 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:33.568 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:33.568 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:33.568 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:15:33.569 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:33.569 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:33.569 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:15:33.569 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:33.570 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:33.570 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:33.570 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:15:33.570 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:33.570 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:33.571 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:33.571 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:15:33.571 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:33.571 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:15:33.572 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:15:33.572 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:15:33.572 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:33.572 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:15:33.573 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.839033_212.1 2021-09-16T20:15:38.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:38.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:38.576 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:15:38.625 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:15:38.625 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:15:38.697 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:15:38.698 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:15:38.698 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:15:38.698 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:15:38.699 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:15:38.699 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:15:38.699 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:38.699 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:38.700 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:15:38.700 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:38.700 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:15:38.701 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:38.701 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:38.701 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:15:38.702 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:38.702 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:15:38.702 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:38.703 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:15:38.703 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:38.703 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:38.703 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:38.703 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:38.704 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:15:38.704 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:38.704 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:38.704 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:15:38.705 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:38.705 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:38.705 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:38.706 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:15:38.706 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:38.706 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:38.707 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:38.707 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:38.707 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:15:38.707 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:38.708 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:38.708 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:15:38.709 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:38.709 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:38.709 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:38.710 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:15:38.710 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:38.710 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:38.710 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:38.711 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:38.711 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:15:38.711 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:38.711 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:38.712 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:15:38.712 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:38.712 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:38.712 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:38.713 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:15:38.713 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:38.713 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:38.714 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:38.714 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:15:38.714 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:38.715 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:15:38.715 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:15:38.715 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:15:38.716 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:38.716 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:15:38.718 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:15:43.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:43.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:43.720 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:15:43.802 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:15:43.802 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:15:43.951 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:15:43.951 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:15:43.951 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:15:43.952 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:15:43.952 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:15:43.952 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:15:43.952 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:43.952 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:43.953 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:15:43.953 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:43.953 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:15:43.953 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:43.954 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:43.954 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:15:43.954 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:43.954 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:15:43.955 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:43.955 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:15:43.955 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:43.955 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:43.955 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:43.956 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:43.956 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:15:43.956 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:43.956 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:43.956 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:15:43.957 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:43.957 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:43.957 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:43.957 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:15:43.958 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:43.958 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:43.958 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:43.958 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:43.958 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:15:43.959 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:43.959 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:43.959 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.971080_217.1", 2021-09-16T20:15:43.959 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:43.959 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:43.960 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:43.960 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.971080Z", 2021-09-16T20:15:43.960 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:43.960 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:43.960 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:43.961 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:43.961 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:15:43.961 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:43.961 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:43.961 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:15:43.962 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:43.962 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:43.962 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:43.962 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:15:43.963 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:43.963 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:43.963 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:43.963 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:15:43.963 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:43.964 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:15:43.964 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:15:43.964 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:15:43.964 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:43.964 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:15:43.966 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.971080_217.1 2021-09-16T20:15:48.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:48.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:48.969 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:15:49.011 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:15:49.011 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:15:49.086 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:15:49.086 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:15:49.086 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:15:49.087 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:15:49.087 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:15:49.087 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:15:49.087 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:49.087 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:49.088 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:15:49.088 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:49.088 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:15:49.088 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:49.089 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:49.089 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:15:49.089 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:49.089 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:15:49.089 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:49.090 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:15:49.090 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:49.090 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:49.090 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:49.091 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:49.091 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:15:49.091 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:49.091 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:49.092 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:15:49.092 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:49.092 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:49.092 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:49.093 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:15:49.093 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:49.093 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:49.093 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:49.094 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:49.094 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:15:49.094 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:49.094 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:49.094 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.895465_215.1", 2021-09-16T20:15:49.095 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:49.095 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:49.095 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:49.095 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.895465Z", 2021-09-16T20:15:49.096 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:49.096 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:49.096 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:49.096 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:49.097 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:15:49.097 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:49.097 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:49.097 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:15:49.097 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:49.098 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:49.098 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:49.098 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:15:49.098 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:49.099 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:49.099 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:49.099 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:15:49.099 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:49.099 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:15:49.100 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:15:49.100 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:15:49.100 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:49.100 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:15:49.101 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.895465_215.1 2021-09-16T20:15:54.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:54.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:54.104 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:15:54.149 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:15:54.150 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:15:54.203 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:15:54.203 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:15:54.203 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:15:54.203 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:15:54.204 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:15:54.204 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:15:54.204 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:54.204 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:54.204 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:15:54.205 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:54.205 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:15:54.205 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:54.205 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:54.206 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:15:54.206 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:54.206 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:15:54.206 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:54.206 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:15:54.207 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:54.207 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:54.207 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:54.207 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:54.208 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:15:54.208 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:54.208 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:54.208 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:15:54.208 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:54.209 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:54.209 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:54.209 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:15:54.209 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:54.209 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:54.210 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:54.210 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:54.210 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:15:54.210 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:54.211 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:54.211 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.841894_213.1", 2021-09-16T20:15:54.211 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:54.211 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:54.211 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:54.212 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.841894Z", 2021-09-16T20:15:54.212 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:54.212 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:54.212 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:54.213 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:54.213 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:15:54.213 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:54.213 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:54.214 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:15:54.214 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:54.214 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:54.214 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:54.215 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:15:54.215 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:54.215 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:54.215 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:54.215 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:15:54.216 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:54.216 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:15:54.216 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:15:54.216 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:15:54.217 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:54.217 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:15:54.218 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.841894_213.1 2021-09-16T20:15:59.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:59.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:15:59.221 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:15:59.270 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:15:59.270 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:15:59.333 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:15:59.334 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:15:59.334 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:15:59.334 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:15:59.334 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:15:59.335 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:15:59.335 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:59.335 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:59.335 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:15:59.336 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:59.336 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:15:59.336 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:59.336 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:59.337 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:15:59.337 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:59.337 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:15:59.337 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:59.338 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:15:59.338 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:59.338 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:59.338 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:59.339 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:59.339 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:15:59.339 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:59.339 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:59.340 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:15:59.340 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:59.340 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:59.340 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:59.341 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:15:59.341 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:59.341 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:59.341 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:59.342 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:59.342 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:15:59.342 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:59.342 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:59.343 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821519.016744_220.1", 2021-09-16T20:15:59.343 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:59.343 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:59.343 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:59.343 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:19.016744Z", 2021-09-16T20:15:59.344 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:59.344 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:59.344 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:59.344 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:15:59.344 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:15:59.345 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:15:59.345 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:15:59.345 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:15:59.345 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:15:59.345 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:15:59.346 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:15:59.346 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:15:59.346 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:15:59.346 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:59.347 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:59.347 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:15:59.347 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:15:59.347 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:15:59.348 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:15:59.348 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:15:59.348 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:15:59.348 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:15:59.349 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821519.016744_220.1 2021-09-16T20:16:04.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:04.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:04.352 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:16:04.430 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:16:04.430 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:16:04.636 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:16:04.637 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:16:04.637 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:16:04.637 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:16:04.637 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:16:04.638 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:16:04.638 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:04.638 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:04.638 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:16:04.639 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:04.639 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:16:04.639 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:04.639 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:04.639 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:16:04.640 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:04.640 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:16:04.640 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:04.640 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:16:04.641 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:04.641 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:04.641 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:04.641 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:04.641 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:16:04.642 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:04.642 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:04.642 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:16:04.642 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:04.643 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:04.643 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:04.643 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:16:04.643 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:04.643 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:04.644 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:04.644 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:04.644 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:16:04.644 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:04.645 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:04.645 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.839033_212.1", 2021-09-16T20:16:04.645 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:04.645 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:04.645 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:04.646 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.839033Z", 2021-09-16T20:16:04.646 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:04.646 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:04.646 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:04.647 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:04.647 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:16:04.647 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:04.647 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:04.647 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:16:04.648 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:04.648 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:04.648 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:04.648 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:16:04.649 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:04.649 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:04.649 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:04.649 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:16:04.649 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:04.650 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:16:04.650 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:16:04.650 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:16:04.650 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:04.651 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:16:04.652 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.839033_212.1 2021-09-16T20:16:09.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:09.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:09.654 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:16:09.697 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:16:09.697 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:16:09.771 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:16:09.771 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:16:09.771 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:16:09.771 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:16:09.772 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:16:09.772 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:16:09.772 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:09.772 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:09.773 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:16:09.773 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:09.773 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:16:09.773 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:09.773 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:09.774 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:16:09.774 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:09.774 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:16:09.774 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:09.775 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:16:09.775 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:09.775 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:09.775 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:09.775 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:09.776 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:16:09.776 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:09.776 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:09.776 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:16:09.777 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:09.777 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:09.777 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:09.777 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:16:09.777 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:09.778 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:09.778 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:09.778 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:09.778 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:16:09.778 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:09.779 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:09.779 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.791325_211.1", 2021-09-16T20:16:09.779 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:09.779 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:09.780 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:09.780 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.791325Z", 2021-09-16T20:16:09.780 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:09.780 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:09.780 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:09.781 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:09.781 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:16:09.781 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:09.781 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:09.781 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:16:09.781 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:09.782 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:09.782 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:09.782 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:16:09.782 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:09.783 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:09.783 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:09.783 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:16:09.783 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:09.783 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:16:09.783 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:16:09.784 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:16:09.784 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:09.784 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:16:09.785 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.791325_211.1 2021-09-16T20:16:14.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:14.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:14.788 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:16:14.870 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:16:14.870 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:16:15.147 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:16:15.147 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:16:15.147 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:16:15.148 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:16:15.148 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:16:15.148 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:16:15.148 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:15.149 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:15.149 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:16:15.149 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:15.149 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:16:15.150 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:15.150 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:15.150 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:16:15.150 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:15.151 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:16:15.151 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:15.151 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:16:15.151 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:15.151 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:15.152 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:15.152 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:15.152 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:16:15.152 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:15.153 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:15.153 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:16:15.153 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:15.153 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:15.153 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:15.154 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:16:15.154 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:15.154 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:15.154 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:15.155 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:15.155 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:16:15.155 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:15.155 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:15.155 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.839033_212.1", 2021-09-16T20:16:15.156 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:15.156 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:15.156 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:15.156 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.839033Z", 2021-09-16T20:16:15.157 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:15.157 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:15.157 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:15.157 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:15.157 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:16:15.158 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:15.158 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:15.158 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:16:15.158 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:15.158 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:15.159 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:15.159 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:16:15.159 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:15.159 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:15.160 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:15.160 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:16:15.160 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:15.160 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:16:15.160 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:16:15.160 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:16:15.161 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:15.161 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:16:15.162 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.839033_212.1 2021-09-16T20:16:20.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:20.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:20.164 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:16:20.207 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:16:20.207 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:16:20.272 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:16:20.272 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:16:20.272 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:16:20.273 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:16:20.273 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:16:20.273 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:16:20.273 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:20.274 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:20.274 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:16:20.274 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:20.274 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:16:20.275 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:20.275 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:20.275 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:16:20.275 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:20.275 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:16:20.276 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:20.276 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:16:20.276 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:20.276 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:20.277 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:20.277 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:20.277 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:16:20.277 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:20.277 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:20.278 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:16:20.278 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:20.278 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:20.278 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:20.279 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:16:20.279 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:20.279 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:20.279 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:20.279 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:20.280 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:16:20.280 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:20.280 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:20.280 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.839033_212.1", 2021-09-16T20:16:20.280 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:20.281 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:20.281 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:20.281 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.839033Z", 2021-09-16T20:16:20.282 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:20.282 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:20.282 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:20.282 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:20.282 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:16:20.283 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:20.283 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:20.283 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:16:20.283 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:20.284 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:20.284 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:20.284 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:16:20.284 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:20.284 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:20.285 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:20.285 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:16:20.285 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:20.285 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:16:20.286 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:16:20.286 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:16:20.286 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:20.286 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:16:20.287 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.839033_212.1 2021-09-16T20:16:25.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:25.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:25.290 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:16:25.335 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:16:25.335 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:16:25.399 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:16:25.400 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:16:25.400 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:16:25.400 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:16:25.401 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:16:25.401 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:16:25.401 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:25.401 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:25.401 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:16:25.402 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:25.402 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:16:25.402 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:25.402 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:25.403 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:16:25.403 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:25.403 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:16:25.403 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:25.404 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:16:25.404 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:25.404 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:25.404 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:25.405 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:25.405 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:16:25.405 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:25.405 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:25.406 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:16:25.406 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:25.406 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:25.406 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:25.407 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:16:25.407 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:25.407 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:25.407 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:25.408 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:25.408 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:16:25.408 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:25.408 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:25.408 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.768089_209.1", 2021-09-16T20:16:25.409 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:25.409 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:25.409 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:25.409 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.768089Z", 2021-09-16T20:16:25.410 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:25.410 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:25.410 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:25.410 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:25.411 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:16:25.411 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:25.411 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:25.411 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:16:25.411 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:25.412 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:25.412 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:25.412 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:16:25.412 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:25.412 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:25.413 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:25.413 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:16:25.413 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:25.413 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:16:25.414 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:16:25.414 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:16:25.414 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:25.414 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:16:25.416 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.768089_209.1 2021-09-16T20:16:30.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:30.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:30.419 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:16:30.475 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:16:30.475 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:16:30.716 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:16:30.716 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:16:30.716 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:16:30.717 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:16:30.717 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:16:30.717 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:16:30.717 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:30.718 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:30.718 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:16:30.718 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:30.718 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:16:30.718 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:30.719 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:30.719 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:16:30.719 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:30.719 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:16:30.720 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:30.720 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:16:30.720 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:30.720 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:30.721 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:30.721 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:30.721 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:16:30.721 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:30.721 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:30.722 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:16:30.722 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:30.722 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:30.722 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:30.723 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:16:30.723 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:30.723 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:30.723 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:30.724 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:30.724 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:16:30.724 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:30.724 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:30.724 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.839033_212.1", 2021-09-16T20:16:30.725 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:30.725 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:30.725 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:30.725 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.839033Z", 2021-09-16T20:16:30.726 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:30.726 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:30.726 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:30.726 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:30.726 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:16:30.727 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:30.727 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:30.727 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:16:30.727 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:30.728 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:30.728 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:30.728 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:16:30.728 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:30.728 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:30.729 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:30.729 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:16:30.729 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:30.729 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:16:30.729 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:16:30.730 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:16:30.730 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:30.730 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:16:30.731 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.839033_212.1 2021-09-16T20:16:35.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:35.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:35.734 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:16:35.797 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:16:35.797 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:16:35.893 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:16:35.894 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:16:35.894 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:16:35.894 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:16:35.895 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:16:35.895 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:16:35.895 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:35.895 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:35.895 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:16:35.896 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:35.896 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:16:35.896 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:35.896 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:35.897 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:16:35.897 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:35.897 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:16:35.897 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:35.897 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:16:35.898 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:35.898 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:35.898 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:35.898 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:35.898 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:16:35.899 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:35.899 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:35.899 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:16:35.899 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:35.900 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:35.900 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:35.900 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:16:35.900 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:35.900 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:35.901 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:35.901 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:35.901 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:16:35.901 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:35.902 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:35.902 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.788603_210.1", 2021-09-16T20:16:35.902 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:35.902 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:35.902 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:35.903 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.788603Z", 2021-09-16T20:16:35.903 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:35.903 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:35.903 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:35.904 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:35.904 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:16:35.904 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:35.904 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:35.904 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:16:35.905 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:35.905 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:35.905 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:35.905 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:16:35.905 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:35.906 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:35.906 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:35.906 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:16:35.906 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:35.907 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:16:35.907 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:16:35.907 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:16:35.907 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:35.907 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:16:35.909 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.788603_210.1 2021-09-16T20:16:40.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:40.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:40.911 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:16:40.957 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:16:40.957 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:16:41.029 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:16:41.029 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:16:41.029 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:16:41.029 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:16:41.030 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:16:41.030 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:16:41.030 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:41.031 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:41.031 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:16:41.031 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:41.031 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:16:41.031 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:41.032 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:41.032 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:16:41.032 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:41.032 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:16:41.033 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:41.033 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:16:41.033 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:41.033 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:41.034 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:41.034 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:41.034 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:16:41.034 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:41.034 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:41.035 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:16:41.035 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:41.035 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:41.035 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:41.036 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:16:41.036 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:41.036 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:41.037 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:41.037 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:41.037 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:16:41.038 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:41.038 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:41.038 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:16:41.039 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:41.039 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:41.039 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:41.039 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:16:41.040 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:41.040 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:41.040 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:41.041 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:41.041 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:16:41.041 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:41.041 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:41.042 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:16:41.042 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:41.042 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:41.042 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:41.043 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:16:41.043 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:41.043 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:41.043 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:41.043 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:16:41.044 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:41.044 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:16:41.044 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:16:41.044 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:16:41.045 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:41.045 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:16:41.046 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:16:46.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:46.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:46.049 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:16:46.097 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:16:46.097 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:16:46.161 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:16:46.162 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:16:46.162 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:16:46.162 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:16:46.162 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:16:46.163 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:16:46.163 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:46.163 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:46.163 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:16:46.164 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:46.164 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:16:46.164 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:46.164 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:46.164 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:16:46.165 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:46.165 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:16:46.165 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:46.165 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:16:46.165 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:46.166 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:46.166 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:46.166 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:46.166 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:16:46.167 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:46.167 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:46.167 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:16:46.167 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:46.167 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:46.168 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:46.168 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:16:46.168 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:46.168 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:46.168 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:46.169 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:46.169 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:16:46.169 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:46.169 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:46.170 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.791325_211.1", 2021-09-16T20:16:46.170 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:46.170 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:46.170 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:46.170 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.791325Z", 2021-09-16T20:16:46.171 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:46.171 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:46.171 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:46.171 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:46.171 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:16:46.172 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:46.172 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:46.172 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:16:46.172 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:46.173 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:46.173 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:46.173 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:16:46.173 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:46.173 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:46.174 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:46.174 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:16:46.174 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:46.174 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:16:46.174 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:16:46.175 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:16:46.175 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:46.175 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:16:46.176 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.791325_211.1 2021-09-16T20:16:51.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:51.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:51.180 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:16:51.253 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:16:51.254 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:16:51.661 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:16:51.661 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:16:51.661 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:16:51.662 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:16:51.662 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:16:51.662 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:16:51.662 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:51.662 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:51.663 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:16:51.663 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:51.663 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:16:51.663 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:51.664 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:51.664 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:16:51.664 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:51.664 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:16:51.664 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:51.665 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:16:51.665 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:51.665 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:51.665 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:51.666 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:51.666 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:16:51.666 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:51.666 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:51.666 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:16:51.667 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:51.667 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:51.667 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:51.667 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:16:51.668 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:51.668 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:51.668 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:51.668 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:51.668 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:16:51.669 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:51.669 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:51.669 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.839033_212.1", 2021-09-16T20:16:51.669 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:51.669 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:51.670 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:51.670 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.839033Z", 2021-09-16T20:16:51.670 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:51.670 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:51.671 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:51.671 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:51.671 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:16:51.671 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:51.671 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:51.672 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:16:51.672 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:51.672 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:51.672 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:51.673 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:16:51.673 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:51.673 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:51.673 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:51.673 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:16:51.674 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:51.674 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:16:51.674 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:16:51.674 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:16:51.674 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:51.675 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:16:51.676 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.839033_212.1 2021-09-16T20:16:56.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:56.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:16:56.679 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:16:56.748 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:16:56.749 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:16:56.890 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:16:56.890 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:16:56.891 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:16:56.891 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:16:56.891 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:16:56.892 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:16:56.892 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:56.892 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:56.892 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:16:56.892 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:56.893 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:16:56.893 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:56.893 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:56.893 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:16:56.894 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:56.894 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:16:56.894 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:56.894 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:16:56.894 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:56.895 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:56.895 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:56.895 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:56.895 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:16:56.896 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:56.896 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:56.896 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:16:56.896 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:56.897 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:56.897 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:56.897 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:16:56.897 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:56.897 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:56.898 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:56.898 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:56.898 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:16:56.898 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:56.898 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:56.899 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:16:56.899 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:56.899 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:56.899 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:56.900 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:16:56.900 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:56.900 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:56.900 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:56.900 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:16:56.901 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:16:56.901 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:16:56.901 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:16:56.901 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:16:56.902 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:16:56.902 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:16:56.902 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:16:56.902 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:16:56.902 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:16:56.903 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:56.903 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:56.903 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:16:56.903 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:16:56.903 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:16:56.904 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:16:56.904 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:16:56.904 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:16:56.904 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:16:56.906 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:17:01.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:17:01.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:17:01.909 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:17:01.953 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:17:01.953 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:17:02.021 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:17:02.021 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:17:02.022 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:17:02.022 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:17:02.022 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:17:02.022 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:17:02.023 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:02.023 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:02.023 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:17:02.023 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:02.023 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:17:02.024 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:02.024 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:02.024 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:17:02.024 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:02.025 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:17:02.025 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:02.025 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:17:02.025 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:02.025 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:02.026 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:02.026 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:02.026 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:17:02.027 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:02.030 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:02.030 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:17:02.031 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:02.031 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:02.031 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:02.031 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:17:02.032 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:02.032 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:02.032 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:02.032 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:02.033 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:17:02.033 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:02.033 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:02.033 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.768089_209.1", 2021-09-16T20:17:02.033 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:02.034 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:02.034 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:02.034 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.768089Z", 2021-09-16T20:17:02.034 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:02.035 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:02.035 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:02.035 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:02.035 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:17:02.035 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:02.036 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:02.036 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:17:02.036 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:02.036 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:02.036 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:02.037 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:17:02.037 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:02.037 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:02.038 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:02.038 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:17:02.038 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:02.038 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:17:02.038 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:17:02.039 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:17:02.039 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:02.039 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:17:02.040 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.768089_209.1 2021-09-16T20:17:07.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:17:07.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:17:07.043 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:17:07.092 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:17:07.093 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:17:07.175 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:17:07.175 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:17:07.175 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:17:07.176 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:17:07.176 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:17:07.176 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:17:07.176 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:07.176 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:07.177 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:17:07.177 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:07.177 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:17:07.177 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:07.178 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:07.178 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:17:07.178 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:07.178 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:17:07.179 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:07.179 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:17:07.179 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:07.179 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:07.179 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:07.180 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:07.180 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:17:07.180 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:07.180 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:07.181 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:17:07.181 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:07.181 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:07.181 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:07.181 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:17:07.182 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:07.182 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:07.182 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:07.182 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:07.183 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:17:07.183 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:07.183 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:07.183 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.791325_211.1", 2021-09-16T20:17:07.183 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:07.184 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:07.184 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:07.184 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.791325Z", 2021-09-16T20:17:07.184 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:07.185 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:07.185 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:07.185 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:07.185 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:17:07.185 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:07.186 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:07.186 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:17:07.186 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:07.186 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:07.187 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:07.187 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:17:07.187 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:07.187 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:07.187 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:07.188 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:17:07.188 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:07.188 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:17:07.188 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:17:07.189 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:17:07.189 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:07.189 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:17:07.190 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.791325_211.1 2021-09-16T20:17:12.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:17:12.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:17:12.193 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:17:12.241 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:17:12.241 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:17:12.371 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:17:12.372 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:17:12.372 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:17:12.372 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:17:12.372 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:17:12.373 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:17:12.373 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:12.373 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:12.373 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:17:12.374 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:12.374 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:17:12.374 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:12.374 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:12.375 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:17:12.375 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:12.375 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:17:12.375 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:12.375 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:17:12.376 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:12.376 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:12.376 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:12.377 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:12.377 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:17:12.377 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:12.377 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:12.378 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:17:12.378 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:12.378 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:12.378 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:12.378 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:17:12.379 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:12.379 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:12.379 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:12.379 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:12.380 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:17:12.380 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:12.380 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:12.380 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.976537_218.1", 2021-09-16T20:17:12.381 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:12.381 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:12.381 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:12.381 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.976537Z", 2021-09-16T20:17:12.381 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:12.382 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:12.382 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:12.382 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:12.382 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:17:12.383 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:12.383 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:12.383 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:17:12.383 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:12.383 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:12.384 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:12.384 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:17:12.384 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:12.384 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:12.385 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:12.385 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:17:12.385 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:12.385 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:17:12.386 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:17:12.386 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:17:12.386 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:12.386 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:17:12.388 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.976537_218.1 2021-09-16T20:17:17.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:17:17.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:17:17.390 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:17:17.442 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:17:17.442 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:17:17.544 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:17:17.544 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:17:17.545 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:17:17.545 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:17:17.546 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:17:17.546 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:17:17.546 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:17.547 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:17.547 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:17:17.548 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:17.548 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:17:17.549 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:17.549 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:17.549 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:17:17.550 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:17.550 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:17:17.551 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:17.551 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:17:17.552 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:17.552 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:17.553 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:17.553 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:17.554 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:17:17.554 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:17.554 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:17.555 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:17:17.555 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:17.556 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:17.556 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:17.556 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:17:17.557 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:17.557 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:17.558 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:17.558 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:17.558 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:17:17.559 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:17.559 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:17.559 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.791325_211.1", 2021-09-16T20:17:17.559 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:17.560 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:17.560 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:17.560 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.791325Z", 2021-09-16T20:17:17.561 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:17.561 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:17.561 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:17.561 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:17.562 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:17:17.562 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:17.562 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:17.563 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:17:17.563 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:17.563 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:17.564 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:17.564 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:17:17.564 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:17.564 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:17.565 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:17.565 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:17:17.565 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:17.566 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:17:17.566 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:17:17.566 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:17:17.567 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:17.567 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:17:17.569 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.791325_211.1 2021-09-16T20:17:22.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:17:22.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:17:22.573 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:17:22.615 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:17:22.615 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:17:22.674 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:17:22.674 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:17:22.675 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:17:22.675 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:17:22.675 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:17:22.675 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:17:22.676 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:22.676 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:22.676 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:17:22.676 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:22.677 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:17:22.677 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:22.677 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:22.677 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:17:22.677 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:22.678 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:17:22.678 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:22.678 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:17:22.678 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:22.679 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:22.679 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:22.679 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:22.679 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:17:22.679 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:22.680 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:22.680 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:17:22.680 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:22.680 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:22.680 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:22.681 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:17:22.681 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:22.681 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:22.681 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:22.682 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:22.682 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:17:22.682 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:22.682 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:22.682 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.891592_214.1", 2021-09-16T20:17:22.683 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:22.683 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:22.683 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:22.683 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.891592Z", 2021-09-16T20:17:22.684 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:22.684 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:22.684 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:22.684 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:22.684 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:17:22.685 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:22.685 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:22.685 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:17:22.685 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:22.685 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:22.686 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:22.686 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:17:22.686 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:22.686 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:22.687 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:22.687 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:17:22.687 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:22.687 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:17:22.687 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:17:22.688 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:17:22.688 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:22.688 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:17:22.689 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.891592_214.1 2021-09-16T20:17:27.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:17:27.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:17:27.692 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:17:27.741 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:17:27.742 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:17:27.807 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:17:27.807 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": { 2021-09-16T20:17:27.807 INFO:teuthology.orchestra.run.smithi109.stdout: "info": { 2021-09-16T20:17:27.808 INFO:teuthology.orchestra.run.smithi109.stdout: "status": "sync", 2021-09-16T20:17:27.808 INFO:teuthology.orchestra.run.smithi109.stdout: "num_shards": 4, 2021-09-16T20:17:27.808 INFO:teuthology.orchestra.run.smithi109.stdout: "period": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:17:27.808 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:27.809 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:27.809 INFO:teuthology.orchestra.run.smithi109.stdout: "markers": [ 2021-09-16T20:17:27.809 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:27.809 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 0, 2021-09-16T20:17:27.809 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:27.810 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:27.810 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821184.045455_63.1", 2021-09-16T20:17:27.810 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:27.811 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 1, 2021-09-16T20:17:27.811 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:27.811 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:39:44.045455Z", 2021-09-16T20:17:27.811 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:27.811 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:27.812 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:27.812 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:27.812 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 1, 2021-09-16T20:17:27.812 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:27.812 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:27.813 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "", 2021-09-16T20:17:27.813 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:27.813 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:27.813 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:27.814 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "0.000000", 2021-09-16T20:17:27.814 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:27.814 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:27.814 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:27.814 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:27.815 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 2, 2021-09-16T20:17:27.815 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:27.815 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:27.815 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821518.768089_209.1", 2021-09-16T20:17:27.816 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:27.816 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:27.816 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:27.816 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:45:18.768089Z", 2021-09-16T20:17:27.816 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:27.817 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:27.817 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:27.817 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:27.817 INFO:teuthology.orchestra.run.smithi109.stdout: "key": 3, 2021-09-16T20:17:27.818 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:27.818 INFO:teuthology.orchestra.run.smithi109.stdout: "state": 1, 2021-09-16T20:17:27.818 INFO:teuthology.orchestra.run.smithi109.stdout: "marker": "1_1631821337.633408_105.1", 2021-09-16T20:17:27.818 INFO:teuthology.orchestra.run.smithi109.stdout: "next_step_marker": "", 2021-09-16T20:17:27.818 INFO:teuthology.orchestra.run.smithi109.stdout: "total_entries": 0, 2021-09-16T20:17:27.819 INFO:teuthology.orchestra.run.smithi109.stdout: "pos": 0, 2021-09-16T20:17:27.819 INFO:teuthology.orchestra.run.smithi109.stdout: "timestamp": "2021-09-16T19:42:17.633408Z", 2021-09-16T20:17:27.819 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:27.819 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:27.820 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:27.820 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:17:27.820 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:27.820 INFO:teuthology.orchestra.run.smithi109.stdout: "full_sync": { 2021-09-16T20:17:27.820 INFO:teuthology.orchestra.run.smithi109.stdout: "total": 1, 2021-09-16T20:17:27.821 INFO:teuthology.orchestra.run.smithi109.stdout: "complete": 1 2021-09-16T20:17:27.821 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:27.821 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:17:27.822 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1631822766.419765_313.1 target=1_1631821518.768089_209.1 2021-09-16T20:17:32.825 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... FAIL 2021-09-16T20:17:32.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:17:32.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:17:32.827 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-16T20:17:32.875 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:17:32.876 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:17:32.907 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:17:32.907 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "4c01e95e-f6a2-44cb-a0a2-7be0d00a8ae6", 2021-09-16T20:17:32.907 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "remove_zg", 2021-09-16T20:17:32.908 INFO:teuthology.orchestra.run.smithi109.stdout: "api_name": "remove_zg", 2021-09-16T20:17:32.908 INFO:teuthology.orchestra.run.smithi109.stdout: "is_master": "false", 2021-09-16T20:17:32.908 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [], 2021-09-16T20:17:32.908 INFO:teuthology.orchestra.run.smithi109.stdout: "hostnames": [], 2021-09-16T20:17:32.909 INFO:teuthology.orchestra.run.smithi109.stdout: "hostnames_s3website": [], 2021-09-16T20:17:32.909 INFO:teuthology.orchestra.run.smithi109.stdout: "master_zone": "", 2021-09-16T20:17:32.909 INFO:teuthology.orchestra.run.smithi109.stdout: "zones": [], 2021-09-16T20:17:32.909 INFO:teuthology.orchestra.run.smithi109.stdout: "placement_targets": [], 2021-09-16T20:17:32.910 INFO:teuthology.orchestra.run.smithi109.stdout: "default_placement": "", 2021-09-16T20:17:32.910 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T20:17:32.910 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_policy": { 2021-09-16T20:17:32.910 INFO:teuthology.orchestra.run.smithi109.stdout: "groups": [] 2021-09-16T20:17:32.910 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:32.911 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:17:32.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:17:32.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:17:32.913 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-16T20:17:32.956 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:17:32.957 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:17:32.991 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:17:32.991 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "e1923e6b-f0a6-4ea2-b06f-3cf15f08fa34", 2021-09-16T20:17:32.991 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "remove", 2021-09-16T20:17:32.992 INFO:teuthology.orchestra.run.smithi109.stdout: "domain_root": "remove.rgw.meta:root", 2021-09-16T20:17:32.992 INFO:teuthology.orchestra.run.smithi109.stdout: "control_pool": "remove.rgw.control", 2021-09-16T20:17:32.992 INFO:teuthology.orchestra.run.smithi109.stdout: "gc_pool": "remove.rgw.log:gc", 2021-09-16T20:17:32.992 INFO:teuthology.orchestra.run.smithi109.stdout: "lc_pool": "remove.rgw.log:lc", 2021-09-16T20:17:32.993 INFO:teuthology.orchestra.run.smithi109.stdout: "log_pool": "remove.rgw.log", 2021-09-16T20:17:32.993 INFO:teuthology.orchestra.run.smithi109.stdout: "intent_log_pool": "remove.rgw.log:intent", 2021-09-16T20:17:32.993 INFO:teuthology.orchestra.run.smithi109.stdout: "usage_log_pool": "remove.rgw.log:usage", 2021-09-16T20:17:32.993 INFO:teuthology.orchestra.run.smithi109.stdout: "roles_pool": "remove.rgw.meta:roles", 2021-09-16T20:17:32.993 INFO:teuthology.orchestra.run.smithi109.stdout: "reshard_pool": "remove.rgw.log:reshard", 2021-09-16T20:17:32.994 INFO:teuthology.orchestra.run.smithi109.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2021-09-16T20:17:32.994 INFO:teuthology.orchestra.run.smithi109.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2021-09-16T20:17:32.994 INFO:teuthology.orchestra.run.smithi109.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2021-09-16T20:17:32.994 INFO:teuthology.orchestra.run.smithi109.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2021-09-16T20:17:32.995 INFO:teuthology.orchestra.run.smithi109.stdout: "otp_pool": "remove.rgw.otp", 2021-09-16T20:17:32.995 INFO:teuthology.orchestra.run.smithi109.stdout: "system_key": { 2021-09-16T20:17:32.995 INFO:teuthology.orchestra.run.smithi109.stdout: "access_key": "", 2021-09-16T20:17:32.995 INFO:teuthology.orchestra.run.smithi109.stdout: "secret_key": "" 2021-09-16T20:17:32.995 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:32.996 INFO:teuthology.orchestra.run.smithi109.stdout: "placement_pools": [ 2021-09-16T20:17:32.996 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:32.996 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "default-placement", 2021-09-16T20:17:32.996 INFO:teuthology.orchestra.run.smithi109.stdout: "val": { 2021-09-16T20:17:32.997 INFO:teuthology.orchestra.run.smithi109.stdout: "index_pool": "remove.rgw.buckets.index", 2021-09-16T20:17:32.997 INFO:teuthology.orchestra.run.smithi109.stdout: "storage_classes": { 2021-09-16T20:17:32.997 INFO:teuthology.orchestra.run.smithi109.stdout: "STANDARD": { 2021-09-16T20:17:32.997 INFO:teuthology.orchestra.run.smithi109.stdout: "data_pool": "remove.rgw.buckets.data" 2021-09-16T20:17:32.997 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:32.998 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:32.998 INFO:teuthology.orchestra.run.smithi109.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2021-09-16T20:17:32.998 INFO:teuthology.orchestra.run.smithi109.stdout: "index_type": 0 2021-09-16T20:17:32.998 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:32.998 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:32.999 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:32.999 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T20:17:32.999 INFO:teuthology.orchestra.run.smithi109.stdout: "notif_pool": "remove.rgw.log:notif" 2021-09-16T20:17:32.999 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:17:33.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:17:33.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:17:33.002 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-16T20:17:33.082 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:17:33.083 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:17:33.177 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:17:33.178 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:17:33.178 INFO:teuthology.orchestra.run.smithi109.stdout: "epoch": 6, 2021-09-16T20:17:33.178 INFO:teuthology.orchestra.run.smithi109.stdout: "predecessor_uuid": "6be2ae27-983c-4bba-b75f-41d101865b26", 2021-09-16T20:17:33.179 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": [], 2021-09-16T20:17:33.179 INFO:teuthology.orchestra.run.smithi109.stdout: "period_map": { 2021-09-16T20:17:33.179 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:17:33.180 INFO:teuthology.orchestra.run.smithi109.stdout: "zonegroups": [ 2021-09-16T20:17:33.180 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.180 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "4c01e95e-f6a2-44cb-a0a2-7be0d00a8ae6", 2021-09-16T20:17:33.180 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "remove_zg", 2021-09-16T20:17:33.181 INFO:teuthology.orchestra.run.smithi109.stdout: "api_name": "remove_zg", 2021-09-16T20:17:33.181 INFO:teuthology.orchestra.run.smithi109.stdout: "is_master": "false", 2021-09-16T20:17:33.181 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [], 2021-09-16T20:17:33.181 INFO:teuthology.orchestra.run.smithi109.stdout: "hostnames": [], 2021-09-16T20:17:33.181 INFO:teuthology.orchestra.run.smithi109.stdout: "hostnames_s3website": [], 2021-09-16T20:17:33.182 INFO:teuthology.orchestra.run.smithi109.stdout: "master_zone": "e1923e6b-f0a6-4ea2-b06f-3cf15f08fa34", 2021-09-16T20:17:33.182 INFO:teuthology.orchestra.run.smithi109.stdout: "zones": [ 2021-09-16T20:17:33.182 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.182 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "e1923e6b-f0a6-4ea2-b06f-3cf15f08fa34", 2021-09-16T20:17:33.183 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "remove", 2021-09-16T20:17:33.183 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [], 2021-09-16T20:17:33.183 INFO:teuthology.orchestra.run.smithi109.stdout: "log_meta": "false", 2021-09-16T20:17:33.183 INFO:teuthology.orchestra.run.smithi109.stdout: "log_data": "false", 2021-09-16T20:17:33.184 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_index_max_shards": 11, 2021-09-16T20:17:33.184 INFO:teuthology.orchestra.run.smithi109.stdout: "read_only": "false", 2021-09-16T20:17:33.184 INFO:teuthology.orchestra.run.smithi109.stdout: "tier_type": "", 2021-09-16T20:17:33.184 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from_all": "true", 2021-09-16T20:17:33.184 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from": [], 2021-09-16T20:17:33.185 INFO:teuthology.orchestra.run.smithi109.stdout: "redirect_zone": "" 2021-09-16T20:17:33.185 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:33.185 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.185 INFO:teuthology.orchestra.run.smithi109.stdout: "placement_targets": [ 2021-09-16T20:17:33.186 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.186 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "default-placement", 2021-09-16T20:17:33.186 INFO:teuthology.orchestra.run.smithi109.stdout: "tags": [], 2021-09-16T20:17:33.186 INFO:teuthology.orchestra.run.smithi109.stdout: "storage_classes": [ 2021-09-16T20:17:33.186 INFO:teuthology.orchestra.run.smithi109.stdout: "STANDARD" 2021-09-16T20:17:33.187 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:17:33.187 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:33.187 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.187 INFO:teuthology.orchestra.run.smithi109.stdout: "default_placement": "default-placement", 2021-09-16T20:17:33.188 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T20:17:33.188 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_policy": { 2021-09-16T20:17:33.188 INFO:teuthology.orchestra.run.smithi109.stdout: "groups": [] 2021-09-16T20:17:33.188 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:33.189 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.189 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.189 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "test-zonegroup", 2021-09-16T20:17:33.189 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zonegroup", 2021-09-16T20:17:33.189 INFO:teuthology.orchestra.run.smithi109.stdout: "api_name": "", 2021-09-16T20:17:33.190 INFO:teuthology.orchestra.run.smithi109.stdout: "is_master": "true", 2021-09-16T20:17:33.190 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [ 2021-09-16T20:17:33.190 INFO:teuthology.orchestra.run.smithi109.stdout: "http://smithi109.front.sepia.ceph.com:8000" 2021-09-16T20:17:33.190 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.190 INFO:teuthology.orchestra.run.smithi109.stdout: "hostnames": [], 2021-09-16T20:17:33.191 INFO:teuthology.orchestra.run.smithi109.stdout: "hostnames_s3website": [], 2021-09-16T20:17:33.191 INFO:teuthology.orchestra.run.smithi109.stdout: "master_zone": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T20:17:33.191 INFO:teuthology.orchestra.run.smithi109.stdout: "zones": [ 2021-09-16T20:17:33.191 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.192 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T20:17:33.192 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zone1", 2021-09-16T20:17:33.192 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [ 2021-09-16T20:17:33.192 INFO:teuthology.orchestra.run.smithi109.stdout: "http://smithi109.front.sepia.ceph.com:8000" 2021-09-16T20:17:33.192 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.193 INFO:teuthology.orchestra.run.smithi109.stdout: "log_meta": "false", 2021-09-16T20:17:33.193 INFO:teuthology.orchestra.run.smithi109.stdout: "log_data": "true", 2021-09-16T20:17:33.193 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_index_max_shards": 11, 2021-09-16T20:17:33.193 INFO:teuthology.orchestra.run.smithi109.stdout: "read_only": "false", 2021-09-16T20:17:33.194 INFO:teuthology.orchestra.run.smithi109.stdout: "tier_type": "", 2021-09-16T20:17:33.194 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from_all": "true", 2021-09-16T20:17:33.194 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from": [], 2021-09-16T20:17:33.194 INFO:teuthology.orchestra.run.smithi109.stdout: "redirect_zone": "" 2021-09-16T20:17:33.194 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.195 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.195 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "8df190b8-e36d-4ab9-bca1-30bba5bb5ec5", 2021-09-16T20:17:33.195 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zone3", 2021-09-16T20:17:33.195 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [ 2021-09-16T20:17:33.196 INFO:teuthology.orchestra.run.smithi109.stdout: "http://smithi109.front.sepia.ceph.com:8001" 2021-09-16T20:17:33.196 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.196 INFO:teuthology.orchestra.run.smithi109.stdout: "log_meta": "false", 2021-09-16T20:17:33.196 INFO:teuthology.orchestra.run.smithi109.stdout: "log_data": "true", 2021-09-16T20:17:33.197 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_index_max_shards": 11, 2021-09-16T20:17:33.197 INFO:teuthology.orchestra.run.smithi109.stdout: "read_only": "false", 2021-09-16T20:17:33.197 INFO:teuthology.orchestra.run.smithi109.stdout: "tier_type": "", 2021-09-16T20:17:33.197 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from_all": "true", 2021-09-16T20:17:33.197 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from": [], 2021-09-16T20:17:33.198 INFO:teuthology.orchestra.run.smithi109.stdout: "redirect_zone": "" 2021-09-16T20:17:33.198 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.198 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.198 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "c6342467-0ed0-40c7-a5ab-491a56113d8e", 2021-09-16T20:17:33.199 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zone4", 2021-09-16T20:17:33.199 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [ 2021-09-16T20:17:33.199 INFO:teuthology.orchestra.run.smithi109.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2021-09-16T20:17:33.199 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.199 INFO:teuthology.orchestra.run.smithi109.stdout: "log_meta": "false", 2021-09-16T20:17:33.200 INFO:teuthology.orchestra.run.smithi109.stdout: "log_data": "true", 2021-09-16T20:17:33.200 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_index_max_shards": 11, 2021-09-16T20:17:33.200 INFO:teuthology.orchestra.run.smithi109.stdout: "read_only": "false", 2021-09-16T20:17:33.200 INFO:teuthology.orchestra.run.smithi109.stdout: "tier_type": "pubsub", 2021-09-16T20:17:33.202 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from_all": "false", 2021-09-16T20:17:33.202 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from": [ 2021-09-16T20:17:33.202 INFO:teuthology.orchestra.run.smithi109.stdout: "test-zone1" 2021-09-16T20:17:33.202 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.203 INFO:teuthology.orchestra.run.smithi109.stdout: "redirect_zone": "" 2021-09-16T20:17:33.203 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.203 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.203 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "ceba8a70-b881-4808-8acf-e8ddfa5f6daf", 2021-09-16T20:17:33.204 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zone2", 2021-09-16T20:17:33.204 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [ 2021-09-16T20:17:33.204 INFO:teuthology.orchestra.run.smithi109.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2021-09-16T20:17:33.204 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.204 INFO:teuthology.orchestra.run.smithi109.stdout: "log_meta": "false", 2021-09-16T20:17:33.205 INFO:teuthology.orchestra.run.smithi109.stdout: "log_data": "true", 2021-09-16T20:17:33.205 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_index_max_shards": 11, 2021-09-16T20:17:33.205 INFO:teuthology.orchestra.run.smithi109.stdout: "read_only": "false", 2021-09-16T20:17:33.205 INFO:teuthology.orchestra.run.smithi109.stdout: "tier_type": "", 2021-09-16T20:17:33.206 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from_all": "false", 2021-09-16T20:17:33.206 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from": [], 2021-09-16T20:17:33.206 INFO:teuthology.orchestra.run.smithi109.stdout: "redirect_zone": "" 2021-09-16T20:17:33.206 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:33.207 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.207 INFO:teuthology.orchestra.run.smithi109.stdout: "placement_targets": [ 2021-09-16T20:17:33.207 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.207 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "default-placement", 2021-09-16T20:17:33.207 INFO:teuthology.orchestra.run.smithi109.stdout: "tags": [], 2021-09-16T20:17:33.208 INFO:teuthology.orchestra.run.smithi109.stdout: "storage_classes": [ 2021-09-16T20:17:33.208 INFO:teuthology.orchestra.run.smithi109.stdout: "STANDARD" 2021-09-16T20:17:33.208 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:17:33.208 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:33.209 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.209 INFO:teuthology.orchestra.run.smithi109.stdout: "default_placement": "default-placement", 2021-09-16T20:17:33.209 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T20:17:33.209 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_policy": { 2021-09-16T20:17:33.209 INFO:teuthology.orchestra.run.smithi109.stdout: "groups": [] 2021-09-16T20:17:33.210 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:33.210 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:33.210 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.210 INFO:teuthology.orchestra.run.smithi109.stdout: "short_zone_ids": [ 2021-09-16T20:17:33.211 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.211 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T20:17:33.211 INFO:teuthology.orchestra.run.smithi109.stdout: "val": 3703685179 2021-09-16T20:17:33.211 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.212 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.212 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "8df190b8-e36d-4ab9-bca1-30bba5bb5ec5", 2021-09-16T20:17:33.212 INFO:teuthology.orchestra.run.smithi109.stdout: "val": 3204282247 2021-09-16T20:17:33.212 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.212 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.213 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "c6342467-0ed0-40c7-a5ab-491a56113d8e", 2021-09-16T20:17:33.213 INFO:teuthology.orchestra.run.smithi109.stdout: "val": 1461382917 2021-09-16T20:17:33.213 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.213 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.214 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "ceba8a70-b881-4808-8acf-e8ddfa5f6daf", 2021-09-16T20:17:33.214 INFO:teuthology.orchestra.run.smithi109.stdout: "val": 3287816121 2021-09-16T20:17:33.214 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.214 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.214 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "e1923e6b-f0a6-4ea2-b06f-3cf15f08fa34", 2021-09-16T20:17:33.215 INFO:teuthology.orchestra.run.smithi109.stdout: "val": 2624582520 2021-09-16T20:17:33.215 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:33.215 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:17:33.215 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.216 INFO:teuthology.orchestra.run.smithi109.stdout: "master_zonegroup": "test-zonegroup", 2021-09-16T20:17:33.216 INFO:teuthology.orchestra.run.smithi109.stdout: "master_zone": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T20:17:33.216 INFO:teuthology.orchestra.run.smithi109.stdout: "period_config": { 2021-09-16T20:17:33.216 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_quota": { 2021-09-16T20:17:33.217 INFO:teuthology.orchestra.run.smithi109.stdout: "enabled": false, 2021-09-16T20:17:33.217 INFO:teuthology.orchestra.run.smithi109.stdout: "check_on_raw": false, 2021-09-16T20:17:33.217 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size": -1, 2021-09-16T20:17:33.217 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size_kb": 0, 2021-09-16T20:17:33.217 INFO:teuthology.orchestra.run.smithi109.stdout: "max_objects": -1 2021-09-16T20:17:33.218 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.218 INFO:teuthology.orchestra.run.smithi109.stdout: "user_quota": { 2021-09-16T20:17:33.218 INFO:teuthology.orchestra.run.smithi109.stdout: "enabled": false, 2021-09-16T20:17:33.218 INFO:teuthology.orchestra.run.smithi109.stdout: "check_on_raw": false, 2021-09-16T20:17:33.219 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size": -1, 2021-09-16T20:17:33.219 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size_kb": 0, 2021-09-16T20:17:33.219 INFO:teuthology.orchestra.run.smithi109.stdout: "max_objects": -1 2021-09-16T20:17:33.219 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:33.219 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.220 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T20:17:33.220 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_name": "test-realm", 2021-09-16T20:17:33.220 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:33.220 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:17:33.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:17:33.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:17:33.222 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-16T20:17:33.281 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:17:33.282 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:17:33.315 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T20:17:33.315+0000 7f9cf2a29380 0 zone id e1923e6b-f0a6-4ea2-b06f-3cf15f08fa34 is not a part of zonegroup test-zonegroup 2021-09-16T20:17:33.316 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T20:17:33.315+0000 7f9cf2a29380 0 zone id e1923e6b-f0a6-4ea2-b06f-3cf15f08fa34 is not a part of zonegroup default 2021-09-16T20:17:33.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:17:33.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:17:33.323 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-16T20:17:33.407 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:17:33.408 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:17:33.446 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T20:17:33.446+0000 7f1b63e48380 0 ERROR: zonegroup remove_zg has a non existent master zone 2021-09-16T20:17:33.451 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-09-16T20:17:33.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:17:33.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:17:33.454 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-16T20:17:33.503 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:17:33.503 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:17:33.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:17:33.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-16T20:17:33.551 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-16T20:17:33.610 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:17:33.610 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:17:33.777 INFO:teuthology.orchestra.run.smithi109.stdout:{ 2021-09-16T20:17:33.778 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:17:33.778 INFO:teuthology.orchestra.run.smithi109.stdout: "epoch": 7, 2021-09-16T20:17:33.778 INFO:teuthology.orchestra.run.smithi109.stdout: "predecessor_uuid": "6be2ae27-983c-4bba-b75f-41d101865b26", 2021-09-16T20:17:33.779 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_status": [], 2021-09-16T20:17:33.779 INFO:teuthology.orchestra.run.smithi109.stdout: "period_map": { 2021-09-16T20:17:33.779 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "db0b3d90-59a0-4d0b-99ed-8408dd9cf54f", 2021-09-16T20:17:33.779 INFO:teuthology.orchestra.run.smithi109.stdout: "zonegroups": [ 2021-09-16T20:17:33.779 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.780 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "test-zonegroup", 2021-09-16T20:17:33.780 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zonegroup", 2021-09-16T20:17:33.780 INFO:teuthology.orchestra.run.smithi109.stdout: "api_name": "", 2021-09-16T20:17:33.780 INFO:teuthology.orchestra.run.smithi109.stdout: "is_master": "true", 2021-09-16T20:17:33.781 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [ 2021-09-16T20:17:33.781 INFO:teuthology.orchestra.run.smithi109.stdout: "http://smithi109.front.sepia.ceph.com:8000" 2021-09-16T20:17:33.781 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.781 INFO:teuthology.orchestra.run.smithi109.stdout: "hostnames": [], 2021-09-16T20:17:33.782 INFO:teuthology.orchestra.run.smithi109.stdout: "hostnames_s3website": [], 2021-09-16T20:17:33.782 INFO:teuthology.orchestra.run.smithi109.stdout: "master_zone": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T20:17:33.782 INFO:teuthology.orchestra.run.smithi109.stdout: "zones": [ 2021-09-16T20:17:33.782 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.783 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T20:17:33.783 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zone1", 2021-09-16T20:17:33.783 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [ 2021-09-16T20:17:33.783 INFO:teuthology.orchestra.run.smithi109.stdout: "http://smithi109.front.sepia.ceph.com:8000" 2021-09-16T20:17:33.784 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.784 INFO:teuthology.orchestra.run.smithi109.stdout: "log_meta": "false", 2021-09-16T20:17:33.784 INFO:teuthology.orchestra.run.smithi109.stdout: "log_data": "true", 2021-09-16T20:17:33.784 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_index_max_shards": 11, 2021-09-16T20:17:33.784 INFO:teuthology.orchestra.run.smithi109.stdout: "read_only": "false", 2021-09-16T20:17:33.785 INFO:teuthology.orchestra.run.smithi109.stdout: "tier_type": "", 2021-09-16T20:17:33.785 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from_all": "true", 2021-09-16T20:17:33.785 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from": [], 2021-09-16T20:17:33.785 INFO:teuthology.orchestra.run.smithi109.stdout: "redirect_zone": "" 2021-09-16T20:17:33.786 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.786 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.786 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "8df190b8-e36d-4ab9-bca1-30bba5bb5ec5", 2021-09-16T20:17:33.786 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zone3", 2021-09-16T20:17:33.786 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [ 2021-09-16T20:17:33.787 INFO:teuthology.orchestra.run.smithi109.stdout: "http://smithi109.front.sepia.ceph.com:8001" 2021-09-16T20:17:33.787 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.787 INFO:teuthology.orchestra.run.smithi109.stdout: "log_meta": "false", 2021-09-16T20:17:33.787 INFO:teuthology.orchestra.run.smithi109.stdout: "log_data": "true", 2021-09-16T20:17:33.788 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_index_max_shards": 11, 2021-09-16T20:17:33.788 INFO:teuthology.orchestra.run.smithi109.stdout: "read_only": "false", 2021-09-16T20:17:33.788 INFO:teuthology.orchestra.run.smithi109.stdout: "tier_type": "", 2021-09-16T20:17:33.788 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from_all": "true", 2021-09-16T20:17:33.788 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from": [], 2021-09-16T20:17:33.788 INFO:teuthology.orchestra.run.smithi109.stdout: "redirect_zone": "" 2021-09-16T20:17:33.789 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.789 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.789 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "c6342467-0ed0-40c7-a5ab-491a56113d8e", 2021-09-16T20:17:33.789 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zone4", 2021-09-16T20:17:33.790 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [ 2021-09-16T20:17:33.790 INFO:teuthology.orchestra.run.smithi109.stdout: "http://smithi118.front.sepia.ceph.com:8001" 2021-09-16T20:17:33.790 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.790 INFO:teuthology.orchestra.run.smithi109.stdout: "log_meta": "false", 2021-09-16T20:17:33.791 INFO:teuthology.orchestra.run.smithi109.stdout: "log_data": "true", 2021-09-16T20:17:33.791 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_index_max_shards": 11, 2021-09-16T20:17:33.791 INFO:teuthology.orchestra.run.smithi109.stdout: "read_only": "false", 2021-09-16T20:17:33.791 INFO:teuthology.orchestra.run.smithi109.stdout: "tier_type": "pubsub", 2021-09-16T20:17:33.791 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from_all": "false", 2021-09-16T20:17:33.792 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from": [ 2021-09-16T20:17:33.792 INFO:teuthology.orchestra.run.smithi109.stdout: "test-zone1" 2021-09-16T20:17:33.792 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.792 INFO:teuthology.orchestra.run.smithi109.stdout: "redirect_zone": "" 2021-09-16T20:17:33.792 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.793 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.793 INFO:teuthology.orchestra.run.smithi109.stdout: "id": "ceba8a70-b881-4808-8acf-e8ddfa5f6daf", 2021-09-16T20:17:33.793 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "test-zone2", 2021-09-16T20:17:33.793 INFO:teuthology.orchestra.run.smithi109.stdout: "endpoints": [ 2021-09-16T20:17:33.793 INFO:teuthology.orchestra.run.smithi109.stdout: "http://smithi118.front.sepia.ceph.com:8000" 2021-09-16T20:17:33.794 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.794 INFO:teuthology.orchestra.run.smithi109.stdout: "log_meta": "false", 2021-09-16T20:17:33.794 INFO:teuthology.orchestra.run.smithi109.stdout: "log_data": "true", 2021-09-16T20:17:33.794 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_index_max_shards": 11, 2021-09-16T20:17:33.795 INFO:teuthology.orchestra.run.smithi109.stdout: "read_only": "false", 2021-09-16T20:17:33.795 INFO:teuthology.orchestra.run.smithi109.stdout: "tier_type": "", 2021-09-16T20:17:33.795 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from_all": "false", 2021-09-16T20:17:33.795 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_from": [], 2021-09-16T20:17:33.795 INFO:teuthology.orchestra.run.smithi109.stdout: "redirect_zone": "" 2021-09-16T20:17:33.796 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:33.796 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.796 INFO:teuthology.orchestra.run.smithi109.stdout: "placement_targets": [ 2021-09-16T20:17:33.796 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.796 INFO:teuthology.orchestra.run.smithi109.stdout: "name": "default-placement", 2021-09-16T20:17:33.797 INFO:teuthology.orchestra.run.smithi109.stdout: "tags": [], 2021-09-16T20:17:33.797 INFO:teuthology.orchestra.run.smithi109.stdout: "storage_classes": [ 2021-09-16T20:17:33.797 INFO:teuthology.orchestra.run.smithi109.stdout: "STANDARD" 2021-09-16T20:17:33.797 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:17:33.797 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:33.798 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.798 INFO:teuthology.orchestra.run.smithi109.stdout: "default_placement": "default-placement", 2021-09-16T20:17:33.798 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T20:17:33.798 INFO:teuthology.orchestra.run.smithi109.stdout: "sync_policy": { 2021-09-16T20:17:33.799 INFO:teuthology.orchestra.run.smithi109.stdout: "groups": [] 2021-09-16T20:17:33.799 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:33.799 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:33.799 INFO:teuthology.orchestra.run.smithi109.stdout: ], 2021-09-16T20:17:33.799 INFO:teuthology.orchestra.run.smithi109.stdout: "short_zone_ids": [ 2021-09-16T20:17:33.800 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.825 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T20:17:33.825 INFO:teuthology.orchestra.run.smithi109.stdout: "val": 3703685179 2021-09-16T20:17:33.825 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.826 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.826 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "8df190b8-e36d-4ab9-bca1-30bba5bb5ec5", 2021-09-16T20:17:33.826 INFO:teuthology.orchestra.run.smithi109.stdout: "val": 3204282247 2021-09-16T20:17:33.826 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.827 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.827 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "c6342467-0ed0-40c7-a5ab-491a56113d8e", 2021-09-16T20:17:33.827 INFO:teuthology.orchestra.run.smithi109.stdout: "val": 1461382917 2021-09-16T20:17:33.827 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.828 INFO:teuthology.orchestra.run.smithi109.stdout: { 2021-09-16T20:17:33.828 INFO:teuthology.orchestra.run.smithi109.stdout: "key": "ceba8a70-b881-4808-8acf-e8ddfa5f6daf", 2021-09-16T20:17:33.828 INFO:teuthology.orchestra.run.smithi109.stdout: "val": 3287816121 2021-09-16T20:17:33.828 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:33.828 INFO:teuthology.orchestra.run.smithi109.stdout: ] 2021-09-16T20:17:33.829 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.829 INFO:teuthology.orchestra.run.smithi109.stdout: "master_zonegroup": "test-zonegroup", 2021-09-16T20:17:33.829 INFO:teuthology.orchestra.run.smithi109.stdout: "master_zone": "6395206f-e97e-464f-b7bd-c10485d78bf5", 2021-09-16T20:17:33.829 INFO:teuthology.orchestra.run.smithi109.stdout: "period_config": { 2021-09-16T20:17:33.830 INFO:teuthology.orchestra.run.smithi109.stdout: "bucket_quota": { 2021-09-16T20:17:33.830 INFO:teuthology.orchestra.run.smithi109.stdout: "enabled": false, 2021-09-16T20:17:33.830 INFO:teuthology.orchestra.run.smithi109.stdout: "check_on_raw": false, 2021-09-16T20:17:33.830 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size": -1, 2021-09-16T20:17:33.830 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size_kb": 0, 2021-09-16T20:17:33.831 INFO:teuthology.orchestra.run.smithi109.stdout: "max_objects": -1 2021-09-16T20:17:33.831 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.831 INFO:teuthology.orchestra.run.smithi109.stdout: "user_quota": { 2021-09-16T20:17:33.831 INFO:teuthology.orchestra.run.smithi109.stdout: "enabled": false, 2021-09-16T20:17:33.832 INFO:teuthology.orchestra.run.smithi109.stdout: "check_on_raw": false, 2021-09-16T20:17:33.832 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size": -1, 2021-09-16T20:17:33.832 INFO:teuthology.orchestra.run.smithi109.stdout: "max_size_kb": 0, 2021-09-16T20:17:33.832 INFO:teuthology.orchestra.run.smithi109.stdout: "max_objects": -1 2021-09-16T20:17:33.832 INFO:teuthology.orchestra.run.smithi109.stdout: } 2021-09-16T20:17:33.833 INFO:teuthology.orchestra.run.smithi109.stdout: }, 2021-09-16T20:17:33.833 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_id": "24db56c9-e102-4b9e-a672-6434941d0bb3", 2021-09-16T20:17:33.833 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_name": "test-realm", 2021-09-16T20:17:33.833 INFO:teuthology.orchestra.run.smithi109.stdout: "realm_epoch": 2 2021-09-16T20:17:33.833 INFO:teuthology.orchestra.run.smithi109.stdout:} 2021-09-16T20:17:33.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2021-09-16T20:17:33.836 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-46 2021-09-16T20:17:34.480 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=htrfwo-47 2021-09-16T20:17:34.520 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=htrfwo-48 2021-09-16T20:17:34.589 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-49 2021-09-16T20:17:34.591 INFO:tasks.rgw.c1.client.0.smithi109.stdout:*** Caught signal (Segmentation fault) ** 2021-09-16T20:17:34.591 INFO:tasks.rgw.c1.client.0.smithi109.stdout: in thread 7f4cd9c09700 thread_name:radosgw 2021-09-16T20:17:34.591 INFO:tasks.rgw.c1.client.0.smithi109.stdout: ceph version 17.0.0-7730-gbc749ef4 (bc749ef43279601f3e9305d4ad8723d440abaa66) quincy (dev) 2021-09-16T20:17:34.592 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f4d7779ab20] 2021-09-16T20:17:34.592 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7f4d7a39893e] 2021-09-16T20:17:34.592 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1864) [0x7f4d7a3b66a4] 2021-09-16T20:17:34.592 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 4: /lib64/libradosgw.so.2(+0x534e0d) [0x7f4d7a2e9e0d] 2021-09-16T20:17:34.593 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 5: /lib64/libradosgw.so.2(+0x5363f0) [0x7f4d7a2eb3f0] 2021-09-16T20:17:34.593 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 6: /lib64/libradosgw.so.2(+0x53654c) [0x7f4d7a2eb54c] 2021-09-16T20:17:34.593 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 7: make_fcontext() 2021-09-16T20:17:34.595 INFO:tasks.rgw.c1.client.0.smithi109.stdout:2021-09-16T20:17:34.590+0000 7f4cd9c09700 -1 *** Caught signal (Segmentation fault) ** 2021-09-16T20:17:34.595 INFO:tasks.rgw.c1.client.0.smithi109.stdout: in thread 7f4cd9c09700 thread_name:radosgw 2021-09-16T20:17:34.595 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 2021-09-16T20:17:34.595 INFO:tasks.rgw.c1.client.0.smithi109.stdout: ceph version 17.0.0-7730-gbc749ef4 (bc749ef43279601f3e9305d4ad8723d440abaa66) quincy (dev) 2021-09-16T20:17:34.596 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f4d7779ab20] 2021-09-16T20:17:34.596 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7f4d7a39893e] 2021-09-16T20:17:34.596 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1864) [0x7f4d7a3b66a4] 2021-09-16T20:17:34.596 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 4: /lib64/libradosgw.so.2(+0x534e0d) [0x7f4d7a2e9e0d] 2021-09-16T20:17:34.596 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 5: /lib64/libradosgw.so.2(+0x5363f0) [0x7f4d7a2eb3f0] 2021-09-16T20:17:34.597 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 6: /lib64/libradosgw.so.2(+0x53654c) [0x7f4d7a2eb54c] 2021-09-16T20:17:34.597 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 7: make_fcontext() 2021-09-16T20:17:34.597 INFO:tasks.rgw.c1.client.0.smithi109.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-09-16T20:17:34.597 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 2021-09-16T20:17:34.609 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 0> 2021-09-16T20:17:34.590+0000 7f4cd9c09700 -1 *** Caught signal (Segmentation fault) ** 2021-09-16T20:17:34.610 INFO:tasks.rgw.c1.client.0.smithi109.stdout: in thread 7f4cd9c09700 thread_name:radosgw 2021-09-16T20:17:34.610 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 2021-09-16T20:17:34.610 INFO:tasks.rgw.c1.client.0.smithi109.stdout: ceph version 17.0.0-7730-gbc749ef4 (bc749ef43279601f3e9305d4ad8723d440abaa66) quincy (dev) 2021-09-16T20:17:34.610 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f4d7779ab20] 2021-09-16T20:17:34.610 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7f4d7a39893e] 2021-09-16T20:17:34.611 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1864) [0x7f4d7a3b66a4] 2021-09-16T20:17:34.611 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 4: /lib64/libradosgw.so.2(+0x534e0d) [0x7f4d7a2e9e0d] 2021-09-16T20:17:34.611 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 5: /lib64/libradosgw.so.2(+0x5363f0) [0x7f4d7a2eb3f0] 2021-09-16T20:17:34.611 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 6: /lib64/libradosgw.so.2(+0x53654c) [0x7f4d7a2eb54c] 2021-09-16T20:17:34.612 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 7: make_fcontext() 2021-09-16T20:17:34.612 INFO:tasks.rgw.c1.client.0.smithi109.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-09-16T20:17:34.612 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 2021-09-16T20:17:34.624 INFO:tasks.rgw.c1.client.0.smithi109.stdout: -24> 2021-09-16T20:17:34.590+0000 7f4cd9c09700 -1 *** Caught signal (Segmentation fault) ** 2021-09-16T20:17:34.624 INFO:tasks.rgw.c1.client.0.smithi109.stdout: in thread 7f4cd9c09700 thread_name:radosgw 2021-09-16T20:17:34.624 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 2021-09-16T20:17:34.624 INFO:tasks.rgw.c1.client.0.smithi109.stdout: ceph version 17.0.0-7730-gbc749ef4 (bc749ef43279601f3e9305d4ad8723d440abaa66) quincy (dev) 2021-09-16T20:17:34.625 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f4d7779ab20] 2021-09-16T20:17:34.625 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7f4d7a39893e] 2021-09-16T20:17:34.625 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1864) [0x7f4d7a3b66a4] 2021-09-16T20:17:34.625 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 4: /lib64/libradosgw.so.2(+0x534e0d) [0x7f4d7a2e9e0d] 2021-09-16T20:17:34.626 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 5: /lib64/libradosgw.so.2(+0x5363f0) [0x7f4d7a2eb3f0] 2021-09-16T20:17:34.626 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 6: /lib64/libradosgw.so.2(+0x53654c) [0x7f4d7a2eb54c] 2021-09-16T20:17:34.626 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 7: make_fcontext() 2021-09-16T20:17:34.626 INFO:tasks.rgw.c1.client.0.smithi109.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-09-16T20:17:34.627 INFO:tasks.rgw.c1.client.0.smithi109.stdout: 2021-09-16T20:17:34.718 INFO:tasks.rgw.c1.client.0.smithi109.stderr:daemon-helper: command crashed with signal 11 2021-09-16T20:17:35.681 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-09-16T20:17:41.287 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2021-09-16T20:17:46.892 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~11s 2021-09-16T20:17:52.496 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~17s 2021-09-16T20:17:58.100 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~22s 2021-09-16T20:18:03.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~28s 2021-09-16T20:18:09.310 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~34s 2021-09-16T20:18:14.914 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~39s 2021-09-16T20:18:20.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~45s 2021-09-16T20:18:21.645 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... rgw_multi.tests.test_set_bucket_policy ... ERROR 2021-09-16T20:18:21.646 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-50 2021-09-16T20:18:26.123 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~50s 2021-09-16T20:18:31.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~56s 2021-09-16T20:18:37.332 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~62s 2021-09-16T20:18:42.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~67s 2021-09-16T20:18:48.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~73s 2021-09-16T20:18:54.147 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~78s 2021-09-16T20:18:59.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~84s 2021-09-16T20:19:05.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~90s 2021-09-16T20:19:10.964 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~95s 2021-09-16T20:19:16.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~101s 2021-09-16T20:19:22.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~106s 2021-09-16T20:19:22.709 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ERROR 2021-09-16T20:19:22.710 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-51 2021-09-16T20:19:27.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~112s 2021-09-16T20:19:33.386 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~118s 2021-09-16T20:19:38.990 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~123s 2021-09-16T20:19:44.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~129s 2021-09-16T20:19:50.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~135s 2021-09-16T20:19:55.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~140s 2021-09-16T20:20:01.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~146s 2021-09-16T20:20:07.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~151s 2021-09-16T20:20:08.922 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ERROR 2021-09-16T20:20:08.923 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-52 2021-09-16T20:20:12.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~157s 2021-09-16T20:20:18.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~163s 2021-09-16T20:20:23.831 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~168s 2021-09-16T20:20:29.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~174s 2021-09-16T20:20:35.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~179s 2021-09-16T20:20:40.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~185s 2021-09-16T20:20:46.248 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~191s 2021-09-16T20:20:51.852 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~196s 2021-09-16T20:20:54.959 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ERROR 2021-09-16T20:20:54.960 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-53 2021-09-16T20:20:57.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~202s 2021-09-16T20:21:03.061 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~207s 2021-09-16T20:21:08.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~213s 2021-09-16T20:21:14.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~219s 2021-09-16T20:21:19.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~224s 2021-09-16T20:21:25.479 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~230s 2021-09-16T20:21:31.083 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~235s 2021-09-16T20:21:36.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~241s 2021-09-16T20:21:42.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~247s 2021-09-16T20:21:47.898 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~252s 2021-09-16T20:21:53.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~258s 2021-09-16T20:21:59.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~263s 2021-09-16T20:22:04.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~269s 2021-09-16T20:22:10.316 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~275s 2021-09-16T20:22:13.796 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ERROR 2021-09-16T20:22:13.797 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-54 2021-09-16T20:22:15.921 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~280s 2021-09-16T20:22:21.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~286s 2021-09-16T20:22:27.131 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~291s 2021-09-16T20:22:32.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~297s 2021-09-16T20:22:38.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~303s 2021-09-16T20:22:38.341 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2021-09-16T20:22:38.945 INFO:tasks.ceph.c1.osd.0:Sent signal 15 2021-09-16T20:22:38.947 INFO:tasks.ceph.c1.osd.1:Sent signal 15 2021-09-16T20:22:38.948 INFO:tasks.ceph.c1.osd.2:Sent signal 15 2021-09-16T20:22:38.948 INFO:tasks.ceph.c1.mon.a:Sent signal 15 2021-09-16T20:22:38.948 INFO:tasks.rgw.c1.client.1:Sent signal 15 2021-09-16T20:22:38.949 INFO:tasks.ceph.c1.mgr.x:Sent signal 15 2021-09-16T20:22:38.950 INFO:tasks.ceph.c1.osd.0.smithi109.stderr:2021-09-16T20:22:38.946+0000 7f1828ee5700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 0 (PID: 31399) UID: 0 2021-09-16T20:22:38.950 INFO:tasks.ceph.c1.osd.0.smithi109.stderr:2021-09-16T20:22:38.946+0000 7f1828ee5700 -1 osd.0 60 *** Got signal Terminated *** 2021-09-16T20:22:38.950 INFO:tasks.ceph.c1.osd.0.smithi109.stderr:2021-09-16T20:22:38.946+0000 7f1828ee5700 -1 osd.0 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-16T20:22:38.951 INFO:tasks.ceph.c1.osd.1.smithi109.stderr:2021-09-16T20:22:38.948+0000 7f0e15e29700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 1 (PID: 31401) UID: 0 2021-09-16T20:22:38.951 INFO:tasks.ceph.c1.osd.1.smithi109.stderr:2021-09-16T20:22:38.948+0000 7f0e15e29700 -1 osd.1 60 *** Got signal Terminated *** 2021-09-16T20:22:38.952 INFO:tasks.ceph.c1.osd.1.smithi109.stderr:2021-09-16T20:22:38.948+0000 7f0e15e29700 -1 osd.1 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-16T20:22:38.952 INFO:tasks.ceph.c1.osd.2.smithi109.stderr:2021-09-16T20:22:38.948+0000 7fe627677700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 2 (PID: 31404) UID: 0 2021-09-16T20:22:38.952 INFO:tasks.ceph.c1.osd.2.smithi109.stderr:2021-09-16T20:22:38.948+0000 7fe627677700 -1 osd.2 60 *** Got signal Terminated *** 2021-09-16T20:22:38.953 INFO:tasks.ceph.c1.osd.2.smithi109.stderr:2021-09-16T20:22:38.949+0000 7fe627677700 -1 osd.2 60 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-16T20:22:38.953 INFO:tasks.ceph.c1.mon.a.smithi109.stderr:2021-09-16T20:22:38.949+0000 7fd32169a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster c1 -i a (PID: 31079) UID: 0 2021-09-16T20:22:38.953 INFO:tasks.ceph.c1.mon.a.smithi109.stderr:2021-09-16T20:22:38.949+0000 7fd32169a700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2021-09-16T20:22:38.954 INFO:tasks.rgw.c1.client.1.smithi109.stdout:2021-09-16T20:22:38.949+0000 7f25246b7700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 39716) UID: 0 2021-09-16T20:22:38.954 INFO:tasks.ceph.c1.mgr.x.smithi109.stderr:2021-09-16T20:22:38.949+0000 7f861e3ed700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster c1 -i x (PID: 31083) UID: 0 2021-09-16T20:22:38.955 INFO:tasks.ceph.c1.mgr.x.smithi109.stderr:2021-09-16T20:22:38.949+0000 7f861e3ed700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2021-09-16T20:22:39.032 INFO:tasks.rgw.c1.client.1.smithi109.stdout:2021-09-16T20:22:38.949+0000 7f253555a440 -1 shutting down 2021-09-16T20:23:04.916 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ERROR 2021-09-16T20:23:04.917 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=htrfwo-55 2021-09-16T20:24:28.065 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ERROR 2021-09-16T20:24:28.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:24:28.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T20:24:28.068 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T20:24:28.120 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T20:24:28.121 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T20:29:28.123 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T20:29:28.121+0000 7f459ab4b380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T20:34:28.124 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T20:34:28.122+0000 7f459ab4b380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T20:39:28.125 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T20:39:28.122+0000 7f459ab4b380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T20:44:28.125 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T20:44:28.123+0000 7f459ab4b380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T20:49:28.126 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T20:49:28.123+0000 7f459ab4b380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T20:54:28.126 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T20:54:28.124+0000 7f459ab4b380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T20:59:28.126 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T20:59:28.124+0000 7f459ab4b380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T21:04:28.127 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T21:04:28.125+0000 7f459ab4b380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T21:09:28.128 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T21:09:28.125+0000 7f459ab4b380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T21:14:28.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-16T21:14:28.132 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T21:14:28.127+0000 7f459ab4b380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T21:14:28.132 INFO:teuthology.orchestra.run.smithi109.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-16T21:14:28.134 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... FAIL 2021-09-16T21:14:28.135 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2021-09-16T21:14:28.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T21:14:28.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T21:14:28.138 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T21:14:28.190 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T21:14:28.190 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T21:19:28.192 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T21:19:28.190+0000 7ff7ada49380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T21:24:28.193 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T21:24:28.191+0000 7ff7ada49380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T21:29:28.192 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T21:29:28.191+0000 7ff7ada49380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T21:34:28.192 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T21:34:28.192+0000 7ff7ada49380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T21:39:28.192 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T21:39:28.192+0000 7ff7ada49380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T21:44:28.191 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T21:44:28.193+0000 7ff7ada49380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T21:49:28.191 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T21:49:28.193+0000 7ff7ada49380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T21:54:28.191 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T21:54:28.194+0000 7ff7ada49380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T21:59:28.193 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T21:59:28.194+0000 7ff7ada49380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T22:04:28.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-16T22:04:28.197 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T22:04:28.194+0000 7ff7ada49380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T22:04:28.197 INFO:teuthology.orchestra.run.smithi109.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-16T22:04:28.199 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... FAIL 2021-09-16T22:04:28.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T22:04:28.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T22:04:28.202 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T22:04:28.251 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T22:04:28.252 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T22:09:28.257 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T22:09:28.255+0000 7f8305bc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T22:14:28.258 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T22:14:28.255+0000 7f8305bc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T22:19:28.261 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T22:19:28.257+0000 7f8305bc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T22:24:28.260 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T22:24:28.256+0000 7f8305bc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T22:29:28.260 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T22:29:28.257+0000 7f8305bc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T22:34:28.261 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T22:34:28.257+0000 7f8305bc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T22:39:28.261 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T22:39:28.258+0000 7f8305bc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T22:44:28.263 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T22:44:28.258+0000 7f8305bc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T22:49:28.262 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T22:49:28.259+0000 7f8305bc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T22:54:28.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-16T22:54:28.268 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T22:54:28.259+0000 7f8305bc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T22:54:28.268 INFO:teuthology.orchestra.run.smithi109.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-16T22:54:28.269 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... FAIL 2021-09-16T22:54:28.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T22:54:28.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T22:54:28.273 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T22:54:28.322 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T22:54:28.323 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T22:59:28.324 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T22:59:28.321+0000 7fec7d0ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T23:04:28.325 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T23:04:28.321+0000 7fec7d0ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T23:09:28.325 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T23:09:28.322+0000 7fec7d0ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T23:14:28.327 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T23:14:28.324+0000 7fec7d0ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T23:19:28.328 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T23:19:28.325+0000 7fec7d0ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T23:24:28.328 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T23:24:28.326+0000 7fec7d0ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T23:29:28.330 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T23:29:28.326+0000 7fec7d0ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T23:34:28.330 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T23:34:28.327+0000 7fec7d0ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T23:39:28.330 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T23:39:28.327+0000 7fec7d0ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T23:44:28.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-16T23:44:28.336 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T23:44:28.328+0000 7fec7d0ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T23:44:28.337 INFO:teuthology.orchestra.run.smithi109.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-16T23:44:28.338 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... FAIL 2021-09-16T23:44:28.339 INFO:tasks.rgw_multisite_tests:test s3 notification filter on master ... ok 2021-09-16T23:44:28.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests_ps.test_object_timing ... ok 2021-09-16T23:44:28.340 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2021-09-16T23:44:28.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T23:44:28.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-16T23:44:28.343 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-16T23:44:28.395 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-16T23:44:28.395 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-16T23:49:28.397 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T23:49:28.395+0000 7fe95606a380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T23:54:28.397 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T23:54:28.395+0000 7fe95606a380 0 monclient(hunting): authenticate timed out after 300 2021-09-16T23:59:28.398 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-16T23:59:28.396+0000 7fe95606a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T00:04:28.399 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T00:04:28.396+0000 7fe95606a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T00:09:28.400 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T00:09:28.397+0000 7fe95606a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T00:14:28.400 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T00:14:28.397+0000 7fe95606a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T00:19:28.402 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T00:19:28.398+0000 7fe95606a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T00:24:28.404 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T00:24:28.399+0000 7fe95606a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T00:29:28.402 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T00:29:28.400+0000 7fe95606a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T00:34:28.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T00:34:28.411 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T00:34:28.400+0000 7fe95606a380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T00:34:28.411 INFO:teuthology.orchestra.run.smithi109.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T00:34:28.412 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... FAIL 2021-09-17T00:34:28.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T00:34:28.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T00:34:28.415 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T00:34:28.467 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-17T00:34:28.467 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T00:39:28.470 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T00:39:28.467+0000 7f828ff0d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T00:44:28.470 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T00:44:28.467+0000 7f828ff0d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T00:49:28.470 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T00:49:28.468+0000 7f828ff0d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T00:54:28.474 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T00:54:28.469+0000 7f828ff0d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T00:59:28.471 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T00:59:28.468+0000 7f828ff0d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T01:04:28.475 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T01:04:28.469+0000 7f828ff0d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T01:09:28.473 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T01:09:28.469+0000 7f828ff0d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T01:14:28.477 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T01:14:28.472+0000 7f828ff0d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T01:19:28.475 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T01:19:28.472+0000 7f828ff0d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T01:24:28.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T01:24:28.479 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T01:24:28.473+0000 7f828ff0d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T01:24:28.479 INFO:teuthology.orchestra.run.smithi109.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T01:24:28.480 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... FAIL 2021-09-17T01:24:28.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:24:28.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T01:24:28.483 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T01:24:28.535 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-17T01:24:28.535 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T01:29:28.538 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T01:29:28.535+0000 7f0a38c5d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T01:34:28.538 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T01:34:28.536+0000 7f0a38c5d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T01:39:28.538 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T01:39:28.536+0000 7f0a38c5d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T01:44:28.540 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T01:44:28.537+0000 7f0a38c5d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T01:49:28.540 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T01:49:28.538+0000 7f0a38c5d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T01:54:28.540 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T01:54:28.538+0000 7f0a38c5d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T01:59:28.541 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T01:59:28.539+0000 7f0a38c5d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T02:04:28.541 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T02:04:28.539+0000 7f0a38c5d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T02:09:28.543 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T02:09:28.540+0000 7f0a38c5d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T02:14:28.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T02:14:28.547 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T02:14:28.540+0000 7f0a38c5d380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T02:14:28.547 INFO:teuthology.orchestra.run.smithi109.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T02:14:28.548 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... FAIL 2021-09-17T02:14:28.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:28.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T02:14:28.551 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T02:14:28.600 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-17T02:14:28.601 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T02:19:28.606 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T02:19:28.601+0000 7f64fe652380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T02:24:28.604 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T02:24:28.602+0000 7f64fe652380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T02:29:28.605 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T02:29:28.602+0000 7f64fe652380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T02:34:28.604 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T02:34:28.603+0000 7f64fe652380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T02:39:28.605 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T02:39:28.602+0000 7f64fe652380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T02:44:28.605 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T02:44:28.603+0000 7f64fe652380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T02:49:28.606 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T02:49:28.603+0000 7f64fe652380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T02:54:28.606 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T02:54:28.604+0000 7f64fe652380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T02:59:28.608 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T02:59:28.605+0000 7f64fe652380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:04:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T03:04:28.610 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T03:04:28.605+0000 7f64fe652380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:04:28.610 INFO:teuthology.orchestra.run.smithi109.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T03:04:28.611 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... FAIL 2021-09-17T03:04:28.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T03:04:28.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T03:04:28.614 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T03:04:28.663 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-17T03:04:28.663 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T03:09:28.668 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T03:09:28.664+0000 7f5e54a53380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:14:28.667 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T03:14:28.664+0000 7f5e54a53380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:19:28.668 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T03:19:28.666+0000 7f5e54a53380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:24:28.669 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T03:24:28.666+0000 7f5e54a53380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:29:28.670 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T03:29:28.667+0000 7f5e54a53380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:34:28.670 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T03:34:28.669+0000 7f5e54a53380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:39:28.671 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T03:39:28.669+0000 7f5e54a53380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:44:28.671 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T03:44:28.669+0000 7f5e54a53380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:49:28.672 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T03:49:28.670+0000 7f5e54a53380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:54:28.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T03:54:28.684 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T03:54:28.670+0000 7f5e54a53380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T03:54:28.684 INFO:teuthology.orchestra.run.smithi109.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T03:54:28.685 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... FAIL 2021-09-17T03:54:28.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T03:54:28.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T03:54:28.688 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T03:54:28.738 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-17T03:54:28.739 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T03:59:28.741 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T03:59:28.739+0000 7fe7fcb92380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:04:28.741 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T04:04:28.738+0000 7fe7fcb92380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:09:28.742 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T04:09:28.740+0000 7fe7fcb92380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:14:28.743 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T04:14:28.740+0000 7fe7fcb92380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:19:28.743 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T04:19:28.741+0000 7fe7fcb92380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:24:28.744 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T04:24:28.740+0000 7fe7fcb92380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:29:28.744 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T04:29:28.742+0000 7fe7fcb92380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:34:28.745 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T04:34:28.741+0000 7fe7fcb92380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:39:28.746 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T04:39:28.743+0000 7fe7fcb92380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:44:28.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T04:44:28.757 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T04:44:28.743+0000 7fe7fcb92380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:44:28.758 INFO:teuthology.orchestra.run.smithi109.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T04:44:28.759 INFO:tasks.rgw_multisite_tests:test radosgw-admin commands ... FAIL 2021-09-17T04:44:28.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T04:44:28.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T04:44:28.762 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T04:44:28.811 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-17T04:44:28.812 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T04:49:28.814 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T04:49:28.811+0000 7fba016da380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:54:28.814 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T04:54:28.811+0000 7fba016da380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T04:59:28.814 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T04:59:28.812+0000 7fba016da380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:04:28.815 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T05:04:28.812+0000 7fba016da380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:09:28.817 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T05:09:28.813+0000 7fba016da380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:14:28.817 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T05:14:28.815+0000 7fba016da380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:19:28.817 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T05:19:28.815+0000 7fba016da380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:24:28.817 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T05:24:28.816+0000 7fba016da380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:29:28.817 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T05:29:28.815+0000 7fba016da380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:34:28.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T05:34:28.830 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T05:34:28.817+0000 7fba016da380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:34:28.831 INFO:teuthology.orchestra.run.smithi109.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T05:34:28.832 INFO:tasks.rgw_multisite_tests:test that events are only sent on incremental sync ... FAIL 2021-09-17T05:34:28.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T05:34:28.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T05:34:28.836 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T05:34:28.886 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-17T05:34:28.887 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T05:39:28.889 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T05:39:28.887+0000 7fd33137c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:44:28.889 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T05:44:28.888+0000 7fd33137c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:49:28.890 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T05:49:28.888+0000 7fd33137c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:54:28.890 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T05:54:28.889+0000 7fd33137c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T05:59:28.891 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T05:59:28.889+0000 7fd33137c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:04:28.891 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T06:04:28.890+0000 7fd33137c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:09:28.893 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T06:09:28.891+0000 7fd33137c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:14:28.894 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T06:14:28.891+0000 7fd33137c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:19:28.893 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T06:19:28.891+0000 7fd33137c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:24:28.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T06:24:28.897 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T06:24:28.891+0000 7fd33137c380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:24:28.897 INFO:teuthology.orchestra.run.smithi109.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T06:24:28.899 INFO:tasks.rgw_multisite_tests:test subscriptions for different events ... FAIL 2021-09-17T06:24:28.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T06:24:28.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T06:24:28.901 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T06:24:28.950 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-17T06:24:28.951 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T06:29:28.954 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T06:29:28.950+0000 7f9848274380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:34:28.953 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T06:34:28.950+0000 7f9848274380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:39:28.954 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T06:39:28.950+0000 7f9848274380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:44:28.955 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T06:44:28.951+0000 7f9848274380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:49:28.960 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T06:49:28.952+0000 7f9848274380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:54:28.955 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T06:54:28.952+0000 7f9848274380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T06:59:28.956 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T06:59:28.952+0000 7f9848274380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T07:04:28.956 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T07:04:28.953+0000 7f9848274380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T07:09:28.957 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T07:09:28.953+0000 7f9848274380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T07:14:28.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-17T07:14:29.001 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T07:14:28.955+0000 7f9848274380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T07:14:29.001 INFO:teuthology.orchestra.run.smithi109.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-17T07:14:29.003 INFO:tasks.rgw_multisite_tests:test incremental fetching of events from a subscription ... FAIL 2021-09-17T07:14:29.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T07:14:29.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-17T07:14:29.006 DEBUG:teuthology.orchestra.run.smithi109:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-17T07:14:29.058 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setuser ceph since I am not root 2021-09-17T07:14:29.059 INFO:teuthology.orchestra.run.smithi109.stderr:ignoring --setgroup ceph since I am not root 2021-09-17T07:19:29.061 INFO:teuthology.orchestra.run.smithi109.stderr:2021-09-17T07:19:29.057+0000 7f1ec06a3380 0 monclient(hunting): authenticate timed out after 300 2021-09-17T07:20:51.598 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-09-17T07:20:51.600 DEBUG:teuthology.task.console_log:Killing console logger for smithi109 2021-09-17T07:20:51.601 DEBUG:teuthology.task.console_log:Killing console logger for smithi118 2021-09-17T07:20:51.601 DEBUG:teuthology.exit:Finished running handlers